Artificial intelligence's true capability is no longer determined by how large a model is or how much data trained it; instead, it's now a direct function of how much computing power a user is willing to spend at inference time, according to OpenAI researcher Noam Brown. This fundamental shift means that the traditional way AI labs compare models to each other is becoming obsolete.

Why Are Static Benchmark Comparisons No Longer Reliable?

For years, the AI industry has relied on what Brown calls the "benchmark grid," a simple table showing how different models perform on standardized tests. One model gets a score of 87, another gets 92, and researchers declare the second one superior. But this approach fails to capture a critical reality: newer models aren't necessarily smarter in a fixed sense. Instead, they're often more efficient thinkers, meaning they can reach higher accuracy levels with significantly less computing power than older versions.

The problem becomes even more apparent when you consider the scale of modern reasoning tasks. During the era of GPT-3, giving an older model a $10 million inference budget wouldn't have yielded significantly better answers. Today, the landscape has shifted entirely. A model given $1,000 to think about a complex problem will drastically outperform the same model given just $1 to solve it. Yet current benchmark grids don't account for this variable at all.

"The core shift in AI is the transition from model-centric intelligence to budget-centric intelligence. We must stop asking 'how smart is this model' and start asking 'how smart is this model given a specific amount of compute,'" said Noam Brown.

Noam Brown, Researcher at OpenAI

Brown points out that the industry is stuck in what he calls a "bad equilibrium." Every lab publishes a static grid of benchmark scores to prove their model's superiority, even though researchers know this format is fundamentally flawed. The inertia is powerful; everyone expects a grid because that's what has always been published.

How Much Better Can Models Get If You Give Them More Time to Think?

The practical implications of test-time compute scaling are staggering. Modern models can productively "reason" for weeks or even months on a single problem, making standard evaluation cycles significantly harder to verify. This creates a tension in the model release cycle, where labs are pressured to ship new models quickly even though they haven't had time to fully explore the capabilities that might emerge after extended periods of continuous thinking.

Brown recently used an internal model to disprove the Erdős unit distance conjecture, a famous open problem in mathematics. The fascinating part of this breakthrough is that the capability was already latent in existing models, but it required a specific "scaffold" or steering mechanism to unlock it. Had a researcher been willing to spend $100,000 in inference costs on a general-purpose scaffold months ago, they likely could have achieved the same result before the newer, specialized models were even trained.

This suggests that the ceiling for current AI is much higher than we realize. We simply aren't spending enough time or money on each individual query to see what the models are truly capable of accomplishing. Brown reflects on his PhD work creating poker solvers, noting that while early models struggled with basic game mechanics, current frontier models can optimize complex algorithms by 10x or even 100x efficiency.

How Should AI Labs Evaluate Models Going Forward?

• Add a Compute Axis: Models should be evaluated using an X-axis that measures tokens, cost, or time, rather than a single static Y-axis point. This allows researchers to see how performance scales with additional inference budget.
• Account for Efficiency Gains: When newer models are released, evaluation should control for the "thinking time" used. Newer iterations are often simply more efficient, reaching higher accuracy levels with significantly less compute than older versions, a nuance lost in a single-number score.
• Extend Evaluation Timelines: Because the performance plateau for modern models is now weeks or even months away, labs need to allocate more time for thorough evaluation before release, rather than rushing to publish benchmark grids.

What Are the Safety Implications of Test-Time Compute Scaling?

The shift to budget-centric intelligence creates a dangerous blind spot in current safety policies and preparedness frameworks. These frameworks were largely designed in an era before test-time compute scaling became a dominant paradigm. If a model's dangerous capabilities, such as the ability to assist in creating bioweapons, scale with its inference budget, then a safety evaluation performed on a $10 budget is meaningless for a bad actor willing to spend $1 million.

We are entering a world where "capability" is no longer a fixed trait of the model's weights but a dynamic variable controlled by the user's wallet and patience. Yet current policies largely ignore this inconvenient truth. This creates a critical gap between how labs evaluate models and how adversaries might actually use them in the real world.

Is an "Intelligence Explosion" Coming?

Brown is skeptical of an overnight explosion in AI capabilities, primarily because even if models become smarter, they are still bottlenecked by the physical time required to run long-horizon experiments. Time itself remains the primary constraint for AI labs today, as researchers are limited by how fast they can iterate and evaluate new architectures.

However, Brown does see recursive self-improvement as a significant accelerator for research. The challenge is that models still lack the specific "research taste" required to identify which novel algorithmic directions are worth pursuing. The future of frontier AI labs will be defined by who can most effectively balance rapid scaling with the temporal limitations of rigorous evaluation. As the field moves forward, the ability of models to coordinate, share knowledge, and build on accumulated findings will likely be the next major frontier, moving beyond short context windows toward a more organic, emerging property of collective machine intelligence.

The implications are clear: the industry's evaluation methods must evolve to reflect the reality that AI capability is no longer a fixed number but a scaling function of compute budget. Labs that recognize this shift and adapt their evaluation practices accordingly will have a significant advantage in understanding what their models can truly accomplish.