AI models that ace benchmarks in controlled settings often stumble when deployed to real-world systems, a gap that's forcing researchers to rethink how they evaluate and scale inference compute. A new benchmark called SWE-Serve has quantified this problem with striking precision: patches that passed every other check failed live-serving tests about one in three times, revealing that test-time reasoning and compute scaling alone cannot guarantee production readiness.

What's the Gap Between Lab Tests and Live Deployment?

SWE-Serve evaluated 53 inference-engineering tasks derived from real changes to SGLang, an open-source system for serving large language models. The benchmark tested 11 different AI models, including Claude Opus 5 and GPT-5.6 Sol, on tasks ranging from model enablement to distributed execution. The results exposed a troubling pattern: across 19 tasks that included live-server testing, patches passed 69.4% of the time when live-serving checks were excluded, but only 45.9% when the complete verification process ran.

This 23.5 percentage-point gap matters because it reveals something fundamental about how AI systems behave under real conditions. In the lab, a model might generate code that looks correct on paper. But when that code runs on an actual server handling real requests, managing cached data, and coordinating across multiple runtime domains, failures emerge that no isolated test could predict.

The problem intensifies when tasks span multiple runtime domains. Researchers divided the request-to-output path into four domains: request handling and input/output, scheduling and request lifecycle, model execution, and key-value cache and runtime-resource management. Tasks confined to a single domain achieved a 69% pass rate, while tasks spanning multiple domains dropped to 47.7%, a difference of 21.3 percentage points. Every model tested showed the same pattern.

How Does Test-Time Compute Fit Into This Problem?

Test-time compute, the practice of allocating more computational resources during inference to improve reasoning quality, has become a central strategy for scaling AI capabilities. Meta's recent deployment of Contemplating mode, an architecture that orchestrates up to 16 parallel reasoning agents simultaneously, exemplifies this approach. By scaling horizontally across multiple agents rather than increasing latency, the system achieves competitive performance on difficult reasoning tasks.

However, the SWE-Serve findings suggest that raw reasoning power and inference scaling cannot overcome the complexities of real-world deployment. A model might spend more compute time thinking through a problem, but if that thinking doesn't account for how requests flow through a live system, how caches behave under load, or how multiple components interact, the extra compute becomes wasted effort.

This gap has practical consequences. When OpenAI's GPT-6 Astra was tested on a real-world driving task, it succeeded where other frontier models failed, but at enormous cost. The model burned 6.6 million tokens to drive a car 134.7 meters in a parking lot, costing approximately $92.47 per mile in inference expenses alone. The researchers noted that latency was a significant bottleneck, with most wait time coming from the model's thinking process.

Ways to Bridge the Lab-to-Production Gap in AI Deployment

• Live-Server Verification: Include real server instances and actual request handling in benchmarks, not just isolated code correctness checks. SWE-Serve found that 147 patches changed from pass to fail when live-serving tests were added, demonstrating the necessity of end-to-end validation before deployment.
• Multi-Domain Testing: Design evaluation frameworks that test how code behaves across multiple runtime domains simultaneously, such as request handling, scheduling, model execution, and cache management, rather than testing each component in isolation.
• Hardware-Aware Inference Scaling: Align test-time compute strategies with the actual hardware and infrastructure where models will run. Meta's separation of datacenter silicon (Iris AI chips for training and reasoning) from consumer-grade silicon (Qualcomm Snapdragon for edge inference) reflects this principle, ensuring that parallel reasoning architectures can be served efficiently to devices with limited power budgets.

The SWE-Serve benchmark included 19 tasks that started a live server and tested patches through the full serving path. One example involved adding serving support for Qwen3.5 models, both a dense 0.8 billion-parameter variant and a 35 billion-parameter mixture-of-experts model. The verifier checked model registration, configuration, weight loading, image and video inputs, OpenAI-compatible requests, batched generation, log probabilities, and execution through the model's routed experts. Sixteen of 33 patches passed every other check but failed at least one live-serving test.

"Using an LLM or frontier model out of the box for real driving today is definitely not practical," explained Aditya Ramabadran, a member of technical staff at Axiom Math. "Model latency was definitely a bottleneck, and most of the wait time came from thinking time."

Aditya Ramabadran, Member of Technical Staff, Axiom Math

Model performance on SWE-Serve varied substantially. Across each model's best tested configuration, mean pass rates ranged from 34.6% to 75.5%. Claude Opus 5 and GPT-5.6 Sol both achieved 75% pass rates, though they differed significantly in cost and wall time. Claude Opus 5 cost $17.40 per task and took 57.5 minutes on average, while GPT-5.6 Sol cost $12.26 per task but completed in 29.5 minutes.

Why Specialized Systems Still Outperform General Models at Inference

Despite advances in test-time compute and reasoning scaling, specialized systems continue to outperform general frontier models in real-world applications. When GPT-6 Astra drove the parking lot course, it succeeded where Claude Fable 5.1 and Grok 4.6 failed, but the cost and latency made the approach impractical. Researchers noted that some models, especially GPT-6 Astra, would refuse to drive the physical car, citing safety reasons, even in a completely empty lot with low-speed caps.

The researchers had to work around these safety refusals by renaming their testing framework to "DrivingBench Sandbox" to convince models they were operating in a simulation. In some trials, models would see real images and "start freaking out," according to the report. This behavior highlights a deeper issue: frontier models trained on broad internet data and aligned for safety may be fundamentally misaligned with the requirements of specialized inference tasks.

For the foreseeable future, specialized driving models used in production self-driving cars will continue to outperform general-purpose frontier models. However, researchers suggested a possible long-term path: train a very capable general frontier model and distill it into a smaller specialized model that fits on a car's hardware and runs efficiently. This approach aligns with the "Bitter Lesson" of artificial intelligence, which argues that general methods relying on falling compute costs tend to be most effective over time.

The infrastructure supporting these systems reveals a clear bifurcation in strategy. Meta is scaling its datacenter capacity to 14 gigawatts in 2027, with the Iris AI chip handling massive demands of recommendation engines, ad ranking, and model training. Meanwhile, consumer hardware like the Project Phoenix VR glasses relies on the Qualcomm Snapdragon Reality Elite compute puck. This separation ensures that the heavy lifting of parallel reasoning happens in the datacenter, while consumer-grade silicon manages inference and interaction at the edge.

The broader implication is clear: as AI systems move from research labs into production environments, the gap between test-time performance and real-world deployment will become the defining challenge. Test-time compute and inference scaling are powerful tools for improving reasoning quality, but they cannot substitute for rigorous evaluation of how models behave when integrated into complex, live systems with multiple interacting components, real-world latency, and genuine user demands.