When AI labs publish benchmark scores for new coding agents like Devin, those numbers depend less on the model's true ability than on who ran the test, how they set it up, and which results they chose to report. A single model can score dramatically differently on the same benchmark depending on the testing harness, effort settings, and whether private or public tasks were used, according to a detailed analysis of six major 2026 model launches.

Why Do the Same Benchmarks Show Different Scores Across Labs?

When OpenAI, Anthropic, Google, and other labs announce new models, they lead with benchmark tables that look like direct comparisons. But those numbers hide a four-stage pipeline that shapes the final score before the model's actual capability is even measured. Each stage introduces choices that can shift results significantly.

Consider how Grok 4.5, a mixture-of-experts model from Cursor and SpaceX AI released in July 2026, reported its coding performance. On the Terminal-Bench and SWE-bench Pro benchmarks, Grok's scores were marked as self-reported, meaning Cursor ran the test on its own testing setup rather than using a shared, neutral harness. Even more tellingly, different models in the same row used different effort settings: Grok ran at "high" effort, GPT-5.5 at "xhigh," and Opus 4.8 at "max." None were held constant, making the comparison between rows essentially meaningless.

The problem extends across all major launches. On DeepSWE, a long-running coding benchmark, Kimi K3 scored 67.5 under its own KimiCode harness but 67.3 under the official leaderboard's testing setup. On BrowseComp, the same model scored 91.2 with one context strategy and 90.4 without it. These are not measurement errors; they are legitimate variations caused by how the test was administered.

What Are the Four Stages That Shape a Benchmark Score?

Understanding how benchmark scores are constructed reveals why two labs can report wildly different numbers for the same model on the same test. The pipeline works like this:

• Task Design: The benchmark's authors write the task set and define what counts as a correct answer. For example, Agents' Last Exam, created by UC Berkeley's RDI center, includes 1,490 task instances built from 960 expert-authored workflows across 55 professional subdomains, from finance to manufacturing.
• Setup Selection: The lab picks the testing environment, timeout duration, available tools, and effort level the model runs under. These choices are within the benchmark's rules but can significantly affect performance.
• Grading: The benchmark's authors score the run using their rubric. Some tasks have tiered scoring (Tier 1, Tier 2, Tier 3) where partial credit is possible, and different labs may interpret these tiers differently.
• Result Selection: The lab decides which of several possible result numbers to report. If a model runs multiple times, the lab can report the best run, the average, or a median. This final choice is entirely in the lab's hands.

None of these four stages is part of the model itself, yet each one can move the score while the model stays unchanged.

How Do Private Versus Public Tasks Change What Benchmarks Measure?

A critical distinction in benchmark design is whether tasks are public or private. Agents' Last Exam keeps 90 percent of its tasks private, meaning nobody outside the project can reproduce the reported score. The remaining 10 percent are public and can be audited by anyone, but they also leak into training data over time as models are trained on internet data.

This creates a hidden risk. When evaluators run a model through a public API to test it on private tasks, the model's responses are sent to the servers of the company being graded. Technically, nothing prevents those responses from being logged and used as future training data. The safeguards protecting against this are legal agreements and statistical monitoring for overfitting, not technical barriers.

The consequence is that private benchmarks resist contamination from training data but introduce a trust problem: outside observers cannot verify the results. Public benchmarks can be audited but become less useful over time as models train on them. Labs navigate this tradeoff differently, and their choices affect what the benchmark actually measures.

How to Interpret Benchmark Scores Like a Skeptic

• Check the Harness: Look for footnotes indicating whether the score came from the benchmark's official testing setup or the lab's own harness. Self-reported scores are not necessarily wrong, but they are not directly comparable to scores from other labs' harnesses.
• Verify Effort Settings: If a benchmark row compares multiple models, confirm that all models ran under the same effort level, timeout, and available tools. Different settings make the row a comparison of testing conditions, not model capability.
• Distinguish Public from Private Tasks: A score on private tasks cannot be independently verified, while a score on public tasks can be audited but may reflect training data contamination. Neither is inherently better; they measure different things.
• Look for Footnotes and Caveats: Labs that are transparent about their testing setup and acknowledge limitations in their benchmarks are providing more useful information than those that present a single number without context.

The six major model launches of 2026 illustrate this pattern clearly. GPT-5.6 from OpenAI listed benchmarks across agentic knowledge work, computer use, cybersecurity, science, academics, coding, and abstract reasoning. Claude Fable 5 and Mythos 5 from Anthropic listed about 15 rows across coding, agentic work, and private internal suites from Cognition, Cursor, Hebbia, and IMC. Gemini 3.6 Flash from Google listed 10 rows across coding, computer use, knowledge work, and multimodal tasks. GLM-5.2 from Z.ai, an open-weight model, listed 19 rows and explicitly mixed Z.ai-run evaluations with results from outside evaluators, making it clear that rival columns were not all produced under one controlled setup.

Kimi K3, a 2.8-trillion-parameter mixture-of-experts from Moonshot AI, revealed inconsistencies in its footnotes. Each benchmark ran under one of three harnesses: Moonshot's own KimiCode, Claude Code, or Codex. The rival columns mixed Moonshot's own reruns with scores copied from leaderboards and other labs' posts. Effort settings differed within rows, with Kimi at "max" next to GPT-5.5 at "xhigh".

The takeaway is not that benchmarks are worthless or that labs are being deceptive. Rather, a benchmark score is a measurement of how a model performs under a specific set of conditions, not an absolute measure of capability. The conditions matter as much as the model. When comparing coding agents like Devin to competitors, readers should ask not just "which score is higher?" but "were these scores produced under the same conditions?" The answer, in most cases, is no.