OpenAI's o3 reasoning model has doubled its task-completion horizon roughly every seven months since 2019, reaching around 110 minutes as of March 2025, but independent research shows that occasional success does not equal dependable performance. A new report from IntuitionLabs examining long-running AI agents reveals a critical distinction: a model that completes a difficult task once does not necessarily complete it reliably the tenth time, or across varying conditions. This gap between capability and reliability is reshaping how enterprises think about deploying reasoning models in production.

What's the Difference Between Capability and Reliability?

When AI researchers talk about model performance, they often conflate two separate concepts. Capability measures whether a model can complete a task at all on its best attempt, typically reported as a single success rate. Reliability, by contrast, measures whether the same model completes the same class of task consistently across repeated attempts and across increasing task duration.

A model that scores well on a five-minute version of a task can score far worse on a multi-hour version of the identical task type. This is why single-shot accuracy scores are a poor proxy for production reliability. The IntuitionLabs report examined 14 models across multiple benchmarks and found that mean success rates fell by over 24 percentage points between short and very-long tasks, with software-engineering work degrading faster than document processing.

METR, a nonprofit AI evaluation group, uses a specific metric called the "50% time horizon" to measure this distinction. It is defined as the duration of task, measured in how long the task takes a skilled human, that an agent completes successfully half the time. For OpenAI's o3, that horizon sits at approximately 110 minutes as of the March 2025 study. However, the 80% reliability horizon remains far shorter, showing that occasional long-task success does not mean dependable long-task success.

How Are Enterprises Addressing the Reliability Gap?

Because long-running agents can fail mid-task or produce inconsistent results, enterprises are building reliability engineering into their AI workflows. This includes recovery mechanisms, human oversight, and monitoring systems designed specifically for agents that operate over minutes, hours, or days rather than a single conversational turn.

Recovery depends on durable, replayable state. LangGraph, an open-source framework for building agents, uses checkpointers that persist a thread's graph state as checkpoints for functions including fault tolerance. However, its in-memory saver loses checkpoints when the process restarts, which is why enterprises are investing in persistent storage solutions.

Human-in-the-loop oversight is increasingly treated as a core design feature rather than a stopgap. Anthropic's own telemetry across roughly one million API tool calls found that 73% involved a human in the loop in some form and only 0.8% of actions were irreversible. This suggests that autonomous operation, in practice, requires deliberate human involvement at key decision points.

Steps to Build Reliable Long-Running AI Agents

• Implement Checkpointing: Use persistent state management to save agent progress at regular intervals, enabling work to resume after crashes or pauses without losing context or repeating completed steps.
• Design Human Oversight Into Workflows: Plan for human intervention at critical decision points rather than treating it as an afterthought; research shows 73% of production API calls already involve human review in some form.
• Monitor Task Success, Not Just Uptime: Track whether the agent actually completed its assigned task, not just whether the system is technically running; a running process can still fail its assigned objective.
• Use Duration-Stratified Testing: Test agent performance on short, medium, and very-long versions of the same task type to identify where reliability degrades, rather than relying on single-shot accuracy scores.
• Instrument Execution at the Trace Level: Deploy observability platforms that capture detailed logs of each tool call and decision, enabling root-cause analysis when tasks fail partway through.

What Do Enterprise Surveys Reveal About Agent Adoption?

The reliability gap is creating friction in enterprise adoption. Gartner projects that over 40% of agentic AI projects will be canceled by the end of 2027, while Deloitte finds only 21% of organizations report mature agentic governance despite 74% expecting moderate agent usage by 2027. Capgemini reports that trust in fully autonomous agents fell from 43% to 27% year over year, even as 90% of respondents viewed human involvement as positive or cost-neutral.

These figures suggest that enterprises recognize the value of AI agents but are hesitant to deploy them without human oversight. The gap between capability and reliability is not a temporary problem; it is a structural feature of current reasoning models that requires deliberate architectural choices to address.

No standardized formula for "intervention rate" yet exists across the industry, though researchers have proposed an Autonomy Index defined as the share of task steps completed without human involvement. Enterprise observability platforms from LangChain, Arize, Datadog, Langfuse, Weights and Biases, OpenAI, AWS, and Microsoft all instrument agent execution at the trace level, but none publishes a standardized "uptime" figure for agentic systems, since a technically running agent can still fail its assigned task.

The distinction between capability and reliability will likely define the next phase of AI deployment. Models like OpenAI's o3 have demonstrated impressive capability on reasoning tasks, but enterprises building production systems must account for the fact that occasional success is not the same as dependable performance. This requires investing in recovery mechanisms, human oversight, and monitoring systems designed specifically for the unique challenges of long-running agents.