How Two Checkboxes in the API Tripled GPT-5.6's Results on ARC-AGI-3
OpenAI has published a study showing how changes in testing the GPT-5.6 model affected its performance on ARC-AGI-3.
How Two Checkboxes in the API Tripled GPT-5.6's Results on ARC-AGI-3
Today, OpenAI released an interesting study that shows sometimes it's more important how we test than what we test. But let's go in order.
When GPT-5.6 Sol was released, the results on ARC-AGI-3 for such a powerful model turned out to be suspiciously weak – only 7.8%. This is despite the fact that Sol performed well on Pokémon FireRed and solved, for example, the hypothesis of double coverage of cycles. The team decided to investigate what was going on.
Recall that ARC-AGI-3 is a benchmark for agents. Accordingly, models for interacting with the environment need some kind of wrapper – a harness – that defines what the model sees at each step, how tools are called, what happens with the context, etc. Expectedly, the ARC-AGI-3 environment uses the simplest universal harness to fairly compare models with each other. But for OpenAI, it turned out that the problem was buried in this universality.
Firstly, after each action in the game, the model's entire private reasoning was simply deleted, so Sol did not see its own plans and reasoning from previous steps. Secondly, due to the use of a sliding window, as the history grew, any old actions became invisible to the model.
However, Sol was specifically trained to retain reasoning between steps. The harness disabled this feature, and the results plummeted.
Now, pay attention to the graph above. The green line represents the model's results on the public set of ARC-AGI-3 with the official harness. The blue line represents the harness where they simply enabled the retention of reasoning between calls and replaced the crude truncation of history with context compression (the same settings are used in the production of ChatGPT and Codex). 38.3% versus 13.3%, and at the same time, the token consumption for output decreased by 6 times. For comparison, the average human result is 48%.
The moral from OpenAI: the benchmark measures not only the model but also how it is used. At the very least, it's high time to use the Responses API on such benchmarks instead of the outdated Chat Completions.
Why it matters
AnalysisThis study highlights the importance of testing settings in achieving accurate results in AI benchmarks. Understanding how changes in testing affect outcomes can aid in the further development of AI models.
Discuss in community
Share your questions and insights with developers
Related in Ecosystem
Practical guides, core terminology, and agent skills
Four Resume Tips from OpenAI’s Head of Applied AI Development
Wulfie Bain, OpenAI’s head of applied AI development, shared four resume rules: keep it short, remove irrelevant details, highlight outcomes, and lead with the strongest achievements.
Sam Altman Changes X Account Name to Ram Altman
Sam Altman, CEO of OpenAI, changed the display name of his X account to Ram Altman.
Codex Skills for Beginners: Automating AI Agent Workflows
A comprehensive hands-on guide to building and executing Codex Skills: SKILL.md anatomy, explicit vs. auto-triggering, skills vs. scripts vs. tools, Skills API, and safety boundaries.
JSON — Structured Data for Programs, APIs, and AI Agents
A complete hands-on guide to JSON: syntax specifications, serialization via parse and stringify, REST API consumption, JSON Schema validation, and structured outputs for LLMs.