Hugging Face has introduced a new way to measure whether AI agents can effectively use software libraries, shifting focus from just getting the right answer to how much work it takes to get there. The approach, detailed in a recent blog post, reveals that two AI agents solving the same task can produce identical results through vastly different paths, with one using 40 lines of code and multiple debugging cycles while another completes the task in a single command line call.

Why Does It Matter How an AI Agent Solves a Problem?

For years, benchmarks have measured AI performance by checking the final output. If an agent correctly classified sentiment or transcribed audio, the test passed. But Hugging Face researchers realized this misses a critical dimension: the cost of getting there. When agents work with poorly documented APIs or clunky interfaces, they burn through tokens, take longer, and sometimes fail entirely. This becomes expensive at scale, especially as coding agents increasingly handle real development tasks.

The team used the transformers library, a widely used tool for machine learning tasks, as their case study. They measured not just whether agents succeeded, but how many turns it took, how many tokens they consumed, and whether they followed clean paths or resorted to deprecated workarounds. The results showed that library design directly impacts agent efficiency in measurable ways.

How to Evaluate AI Agent Performance on Your Tools

• Measure the full process: Track not just whether the agent reached the correct answer, but how many steps, tokens, and seconds it took to get there, revealing the true cost of agent interaction with your library.
• Test across multiple access patterns: Evaluate how agents perform with bare pip installations, full source code access, and packaged skill modules that include curated documentation and task-specific examples.
• Run identical hardware comparisons: Execute every test on the same hardware using parallel jobs to ensure fair comparisons across different model versions and tasks without hardware variance skewing results.
• Capture native agent traces: Record the exact commands and decisions each agent made, allowing you to see not just the numbers but the actual path taken, which reveals whether agents used deprecated APIs or found efficient shortcuts.
• Score on multiple axes: Evaluate completion rates, median response time, token usage (both cached and newly generated), error rates, and adoption of tool-specific behavior markers designed into your library.

Hugging Face implemented this harness using open-source models and their own Jobs infrastructure, which allowed them to run thousands of tests in parallel on identical hardware. Every run generates a detailed trace that can be examined afterward, turning raw benchmark numbers into actionable insights about what actually slows agents down.

The team tested three different ways agents could interact with transformers. The first was a bare installation with no additional help. The second gave agents access to the full source code repository. The third provided a packaged "Skill," which included curated documentation and task-specific examples designed for agent readability. Surprisingly, agents sometimes performed better with the full source code than with the carefully curated skill, suggesting that more context isn't always better.

What Does This Mean for Library Developers?

The research reveals two fundamental principles for agent-optimized tooling. First, if something isn't tested for agent use, it doesn't work for agents. Second, if documentation isn't structured for agent discovery, it effectively doesn't exist. This mirrors long-standing software engineering wisdom but applies it to a new audience: not human developers, but AI systems that will interact with your code.

Hugging Face's own command-line interface (CLI) was redesigned with agents in mind, and the results were striking. When agents used the agent-optimized CLI, they consumed 1.3 to 1.8 times fewer tokens on average, and up to 6 times fewer tokens on some tasks. The team wanted to know whether similar improvements could be achieved for transformers, which is why they built this benchmarking harness.

The practical example illustrates the difference clearly. When asked to classify the sentiment of "I absolutely loved the movie, it was fantastic," one agent wrote a 40-line Python script that imported transformers, debugged shape errors, and ran twice before printing the answer. Another agent simply typed a single command line call and got the same result. Both reached the correct answer of "POSITIVE" with 0.9999 confidence, but the paths diverged dramatically in cost and complexity.

This benchmarking approach also helps library maintainers understand which changes actually help agents. When developers add a CLI, improve error messages, or create a Skill, they can now measure whether those changes reduce token consumption and latency. Without this kind of testing, improvements meant for human developers might actually make things harder for agents, sending them down longer, more expensive paths.

The harness captures several key metrics for each run. Match percentage shows whether the final answer was correct. Median time and token counts reveal efficiency. Error rates flag runs that produced no output, preventing silent failures from masquerading as successes. Marker adoption tracks whether agents used tool-specific features designed into the library.

All results land in a live report that anyone can examine directly, with full transparency. Because the system captures the native trace of every agent run, the numbers are just the beginning. Developers and researchers can read exactly what the agent did, command by command, using Hugging Face's agent-traces viewer to understand not just whether something worked, but why.

This shift in how we measure AI agent performance reflects a broader change in software development. As agents become more capable and more integrated into development workflows, the tools they use need to be designed with agent interaction in mind from the start. Clunky APIs and sparse documentation don't just annoy human developers anymore; they create expensive, inefficient paths for AI systems that will increasingly handle real work.