OpenAI publishes the Codex paper and HumanEval

'Evaluating Large Language Models Trained on Code' introduces Codex — the GPT model behind Copilot — and the HumanEval benchmark, where it solves 28.8% of problems (GPT-3: 0%). The paper defines how code models get measured for the next three years.