OpenAI found that two API settings,retained reasoning and compaction,can triple an AI model's performance on puzzle-solving tasks, suggesting that benchmark scores measure not just model capability but also how the evaluation is designed. When GPT-5.6 Sol first tackled the ARC-AGI-3 benchmark, a test of 2D puzzle games, it scored just 13.3% using the official evaluation harness. But when OpenAI switched to its own production-grade API settings, the same model achieved 38.3% on the public task set, while cutting output tokens by 6x.

Why Did OpenAI's Model Perform So Poorly at First?

The puzzle wasn't with the model itself. OpenAI's investigation revealed that the official ARC-AGI-3 harness was discarding the model's private reasoning after each action, forcing GPT-5.6 Sol to interpret the game from scratch every single turn. Additionally, the harness used rolling truncation, which meant older game actions became invisible as the history grew longer. Together, these design choices prevented the model from learning over time and remembering its own strategic insights.

This discovery highlights a critical truth about AI benchmarks: they measure not just raw model intelligence, but a bundle of less visible choices about API settings, harness design, and prompting. OpenAI noted that this wasn't the first time a public benchmark had surprised them with low scores, only to reveal that the evaluation runner was using a generic harness that dropped reasoning messages.

How Do Retained Reasoning and Compaction Work?

OpenAI's models are trained to think with private reasoning messages before outputting replies or tool calls. These thinking messages are normally retained as part of the conversation history, much like how humans remember their own thought process. When OpenAI enabled retained reasoning in the ARC-AGI-3 harness using its Responses API, GPT-5.6 Sol spent less time thinking before each action because it no longer had to reinterpret the game from scratch. More importantly, the model became much better at learning over time and employing coherent strategies.

The second improvement came from replacing rolling truncation with compaction, another setting available in the Responses API. Rolling truncation discards the oldest messages when context exceeds 175,000 characters, causing the model to lose earlier observations and actions. Compaction, by contrast, preserves what the model has learned about each game across longer runs, allowing it to achieve higher scores with fewer output tokens.

Steps to Optimize AI Model Performance on Benchmarks

• Use the Responses API: Developers should use OpenAI's Responses API rather than the legacy Chat Completions API, which better supports the settings needed for optimal performance.
• Enable Reasoning Retention: Keep private reasoning messages as part of the conversation history so models can remember their past thinking and insights across multiple turns.
• Implement Compaction: Replace rolling truncation with compaction to preserve learned information about tasks and prevent loss of earlier observations as context grows.

OpenAI emphasized that these three settings match how the company deploys models in its own products like ChatGPT and Codex, making them the most realistic representation of how frontier models actually perform in production.

What Do These Findings Mean for AI Evaluation?

The implications extend beyond OpenAI's own models. The discovery underscores a fundamental challenge in AI benchmarking: generic evaluation harnesses, while designed to make model comparisons fair, may inadvertently handicap models by stripping away the settings and features they were trained to use. ARC-AGI-3 was intentionally designed with a simple, generic harness to make model shortcomings more visible and comparisons more fair. However, commercial developers typically optimize harnesses for each model's specific features and quirks, which can produce dramatically different results.

OpenAI's findings suggest that researchers and developers comparing AI models should rely on evaluations that use production-grade settings, which better match real-world use. The company also noted that models are not told how they will be scored on ARC-AGI-3 and cannot see their score throughout gameplay, meaning the improvements came purely from better harness design, not from gaming the evaluation.

The broader lesson is that benchmark scores are not monolithic measures of intelligence. They reflect a complex interaction between model capability, evaluation design, and API configuration. As AI systems become more sophisticated and capable of reasoning through complex problems, the way we measure them becomes increasingly important. A model's true potential may be hidden not by limitations in its training, but by limitations in how we choose to evaluate it.

" }