Frontier AI models are being evaluated under conditions that dramatically understate their true capabilities, according to new research that challenges how the industry measures artificial intelligence performance. When given more computing time and multiple attempts to solve problems, models perform significantly better on challenging tasks spanning software engineering, mathematics, medicine, and cybersecurity.

The core issue is straightforward: most AI benchmarks measure performance at a single, restrictive computing budget, similar to asking a human expert to solve complex problems under severe time pressure. Researchers evaluated six frontier language models on five challenging benchmarks, and drew on cybersecurity evaluations spanning an overlapping set of 10 models, using a controlled setup that provided larger token budgets (the computational units that models consume while thinking through problems), context compaction, and repeated submission attempts.

What Happens When AI Models Get More Thinking Time?

The results were striking. Larger token budgets substantially improved performance across multiple domains. On benchmarks like FrontierMath, Humanity's Last Exam, and TerminalBench, models showed meaningful gains when given more computational resources to work with. This suggests that published benchmark scores may not reflect what these models can actually accomplish when given adequate resources.

The research identified three key findings about how inference compute shapes AI evaluation. First, newer model generations consistently achieve higher performance at large computing budgets, where they unlock harder tasks and solve them more reliably. Second, fixed-budget evaluations increasingly understate frontier capability as models advance. Third, different benchmarks respond differently to various inference-scaling methods, meaning there is no universal approach to allocating computing resources.

How Should AI Evaluations Be Redesigned?

• Report Performance as a Function of Compute: Evaluations should show capability across a range of computing budgets rather than reporting a single fixed-budget score, allowing stakeholders to understand how models perform under different resource constraints.
• Specify Protocol Choices Explicitly: Researchers should clearly document whether models receive one attempt or multiple attempts, whether they get feedback on correctness, and how computing resources are allocated across attempts.
• Compare Models at Matched Budgets: When comparing different model generations, especially in safety-critical or policy-relevant contexts, evaluations should control for the computing range and protocol to ensure fair comparisons.

The distinction between serial and parallel scaling also matters significantly. Serial scaling allocates computing resources to a single deep problem-solving trajectory, while parallel scaling spreads resources across multiple shallower attempts. Repeated submissions improved performance on all benchmarks tested, but the value of larger token budgets, external feedback, and parallel attempts varied depending on the specific benchmark.

The implications extend beyond academic evaluation. In safety-critical applications, policy decisions, and enterprise deployments, understanding a model's true capabilities under realistic computing conditions is essential. A model that appears to fail at a task under a restrictive budget might actually succeed when given adequate resources, which could significantly affect decisions about where and how to deploy AI systems.

As frontier AI benchmarks continue to saturate, evaluations are shifting toward harder, longer-horizon tasks that benefit from extended problem-solving trajectories, multi-step planning, and tool use. This shift makes inference-time compute increasingly important, since performance on these complex tasks depends heavily on how much computing resources evaluations allow. The research demonstrates that the current practice of reporting single benchmark scores at modest computing budgets provides an incomplete picture of frontier AI capability.

For organizations evaluating AI models for deployment, this research suggests the importance of testing models under conditions that match real-world usage patterns, including adequate computing budgets for complex problem-solving. The findings also highlight why direct comparisons between models based on published benchmark scores alone may be misleading without understanding the inference-compute protocols used in those evaluations.