OpenAI Shows ARC-AGI-3 Scores Can Shift With Agent Memory Settings
Retained reasoning and context compaction lifted OpenAI's reported public-set result, underscoring that agent benchmarks test the harness as well as the model.
OpenAI published an ARC-AGI-3 experiment on July 29 showing that an agent's benchmark score can change sharply when its execution harness preserves reasoning across actions and compacts older context. The result is important less as another leaderboard claim than as evidence that agent evaluations measure a system around the model, not just the model weights.
What the experiment found
ARC-AGI-3 asks agents to explore unfamiliar two-dimensional games and infer their rules without explicit instructions. OpenAI says GPT-5.6 Sol scored 13.3% on the public set with the benchmark's official generic harness. In an OpenAI Responses API harness that retained reasoning and replaced rolling truncation with compaction, the company reports a 38.3% score and roughly six times fewer output tokens.
The first change preserved private reasoning across tool calls and turns, so the agent did not need to reconstruct its strategy after every game action. The second summarized older context instead of discarding the earliest observations when history became too large. OpenAI's interpretation is that the original setup removed plans and eventually hid past actions, making sustained learning harder.
These are vendor-run results and should not be treated as an independent ranking. OpenAI built the alternative harness around its own API features and recommends that evaluators use settings closer to its production deployments. The experiment did not change the underlying model weights, and it does not establish that every benchmark or every vendor would see the same gain.
Why it matters
Agent benchmarks increasingly test repeated decisions, tool use and long-running context. In that setting, memory policy, truncation, caching and orchestration can materially affect both success and token cost. A neutral-looking harness can still favour simplicity over production realism, while a vendor-tuned harness can favour the model it was designed around.
That creates a reporting challenge: benchmark results should name the model, harness, context policy and tools together. Buyers comparing agents also need to distinguish model capability from system engineering, because the deployed product may perform differently from a bare API evaluation in either direction.
Status
Analysis. The reported scores and harness design come from OpenAI's official experiment. Internal confidence is medium because the setup and measurements are first-party, and the fetched evidence contains no independent reproduction.
Sources
Update note: Last reviewed 2026-07-30. We will revise this post if ARC Prize or another evaluator independently reproduces the harness effect.
Sources
- OpenAI — ARC-AGI-3 harness experiment — official
Drafted with AI assistance from source briefs; reviewed for citation completeness and label accuracy.