A new automated benchmarking framework called STAGE-Claw is reshaping how researchers evaluate whether AI agents can actually complete real-world tasks, moving beyond simple text-based testing to measure actual changes in computer systems. The framework, developed by researchers at the Chinese Academy of Sciences and Meituan, tests large language models (LLMs), which are AI systems trained on vast amounts of text data, by placing them in realistic computing environments and measuring whether they successfully transform initial system states into desired final states.

Why Do Current AI Agent Tests Fall Short?

Existing benchmarks for evaluating AI agents have relied on what researchers call "sandboxed artifacts," meaning they test agents using simplified, isolated environments rather than real applications. For example, some benchmarks evaluate whether an agent can schedule a calendar meeting by checking if it generates a correctly formatted calendar file, rather than verifying that the meeting actually appears in a real calendar application. This approach misses critical real-world challenges like software permissions, tool-access errors, and cross-application consistency.

The limitations of current evaluation methods include several key problems that hinder progress toward reliable personal agents. Researchers identified three major gaps in how AI agents are currently tested:

• Sandboxed Testing Environments: Most benchmarks replace real application state with simplified test files, which ignores software permission errors and tool-access failures that occur in actual user scenarios.
• Manual Task Construction: Existing evaluations typically rely on human-created test cases and fixed question-answer pairs, making it difficult to scale testing across diverse user preferences and evolving contexts.
• Lack of Diagnostic Insight: Traditional evaluation methods only check final results, failing to pinpoint where errors occur within the multi-step completion workflow, such as time-zone conversion mistakes or conflict resolution failures.

How Does STAGE-Claw Change Agent Evaluation?

STAGE-Claw introduces a fundamentally different approach to testing AI agents by shifting from artifact-checking to what researchers call "state-based assessment." Rather than verifying that an agent produces the correct text output, the framework measures whether an agent's actions produce the expected changes in actual system environments. Each benchmark task is formulated as a state-transformation problem, where an agent observes an initial computing environment and must transform it into a target final state.

The framework automates three critical aspects of agent evaluation. First, it automatically generates realistic benchmark tasks from simple task hints, creating complete test scenarios with initial environments, task prompts, ground truth answers, and verification programs. Second, it validates that each generated task is verifiable, appropriately difficult, and reproducible. Third, it evaluates agents by executing them in realistic operating environments and measuring success based on actual system-state changes rather than textual responses alone.

What Did Testing 11 Frontier Models Reveal?

Researchers used STAGE-Claw to build a benchmark containing 40 challenging real-scenario agent tasks grounded in five groups of realistic scenarios. These tasks required agents to perform complex workflows involving cross-source reasoning, tool state updates, and cross-tool consistency. The team then evaluated 11 frontier AI models on this benchmark, analyzing their task scores, costs, tool-call reliability, and common failure patterns.

The evaluation revealed significant insights about how well current AI models perform when required to interact with real computing environments rather than simply generating text. By measuring performance through actual system-state verification rather than final-artifact checking, the framework identified specific failure modes and reliability issues that traditional benchmarks would have missed. These findings contribute critical insights for developing more reliable, state-based, and extensible agent evaluation systems.

How to Implement State-Based Agent Evaluation

• Define State Transformations: Formulate agent tasks as state-transformation problems where success is measured by whether the agent transforms an initial environment into a target final state, rather than by checking text outputs.
• Use Real Computing Environments: Test agents in realistic operating environments with actual applications, permissions, and tool-access constraints rather than simplified sandboxed test files.
• Automate Task Generation: Implement automated frameworks that generate benchmark tasks from task hints and validate their verifiability, difficulty, and reproducibility to enable scalable evaluation across diverse scenarios.
• Implement Process-Aware Diagnostics: Analyze fine-grained metrics throughout the agent's task completion workflow to localize where errors occur, such as in time-zone conversion, conflict resolution, or data reconciliation steps.

The STAGE-Claw framework represents a significant shift in how the AI research community evaluates autonomous agents. By moving beyond text-based testing and sandboxed environments, researchers can now assess whether AI agents can reliably handle the complex, multi-step tasks that users actually need them to perform. The framework's code is publicly available, enabling other researchers to build upon this approach and continue advancing agent evaluation methodology.

This development is particularly significant for the growing ecosystem of personal AI agents that integrate with email, calendars, file systems, browsers, and other everyday applications. As these agents become more prevalent in real-world use, having reliable evaluation methods becomes essential for ensuring they work correctly before deployment. The STAGE-Claw framework provides a scalable, automated approach to this challenge, potentially accelerating progress toward more dependable AI agents that users can trust with their daily computing tasks.