Anthropic's new claude plugin eval command answers a fundamental question plaguing AI development: how do you prove that a plugin actually makes Claude better? Released in Claude Code v2.1.269 on September 11, 2026, the tool lets developers run the same coding tasks with and without a plugin, score the results, and measure the actual improvement rather than relying on subjective impressions or cherry-picked examples.

The problem it solves is deceptively simple but critical. Modern large language models (LLMs), which are AI systems trained on vast amounts of text to understand and generate human language, are already remarkably capable. A plugin can produce an excellent result and still contribute almost nothing if Claude would have produced the same result without it. That distinction matters enormously when deciding whether to ship a plugin, maintain it, or simplify it away.

What Makes This Different From Traditional Software Testing?

The claude plugin eval command introduces what developers call an "ablation test" to AI development. The same test cases run twice: once with the plugin enabled, once without it. The difference between the two scores reveals the plugin's actual contribution. This transforms plugin development from "this prompt seems better" to "this extension improved this benchmark by a measurable amount".

Consider two hypothetical results. In the first scenario, a security review plugin scores 0.92 with the plugin enabled but only 0.17 without it, yielding a delta of positive 0.75. That is a strong signal that the plugin provides meaningful value. In the second scenario, both with and without the plugin score 1.00, meaning the base Claude system already handles the task perfectly. The plugin may be unnecessary for that case.

This leads to a practical framework for interpreting results. When a plugin shows high performance with the plugin but low performance without it, developers should keep it and protect the behavior with regression tests. When both scores are high, the base model may already handle the task, suggesting the plugin instructions could be simplified or removed. When both scores are low, the plugin itself needs redesign. And if performance is lower with the plugin than without it, something is actively harming the result, whether that is conflicting constraints, tool calls, context pollution, or bad routing logic.

How to Set Up and Run Plugin Evaluations

• Organize your test structure: Create a plugin directory with a.claude-plugin folder containing plugin.json, a skills folder with your plugin code, and an evals folder with numbered test cases, each containing a prompt.md file and a graders subdirectory with evaluation criteria.
• Design comprehensive test cases: Include core success cases that prove the main capability, difficult edge cases that test deeper reasoning, ambiguous requests that test clarification behavior, should-fire cases that confirm the plugin activates when needed, should-not-fire cases that verify it does not activate on unrelated requests, negative controls that confirm the plugin does not add unnecessary noise, regression tests that protect previously fixed failures, and cost or latency tests that detect inefficient behavior.
• Run a low-cost pilot first: Before paying for a large benchmark, confirm every option with the --help flag in your installed Claude Code version, then run the smallest useful suite. Current build-inspection references describe a single-repetition pilot using a command like claude plugin eval. --runs 1 --ablation with-without --no-scaffold --no-publish.

The inclusion of should-not-fire cases is particularly important. A plugin that activates on every vaguely related request may look powerful in demos while degrading the overall product through unnecessary context, token usage, tool calls, or over-constrained answers. Testing what the plugin should not do is as critical as testing what it should.

Why Stability Matters as Much as Quality?

A strong evaluation suite measures both quality and stability. LLM behavior is probabilistic, meaning a single successful run does not prove reliability. The same prompt may produce slightly different outputs on different runs, which is why repeated testing matters.

The right way to test an AI plugin is to evaluate behavior, not exact text. Instead of expecting one precise answer, a better evaluation asks whether the important outcome happened. For a security review plugin, that means asking: Was the vulnerability found? Was its severity reasonable? Was the vulnerable data flow explained? Was a concrete remediation proposed? Did the plugin avoid unrelated changes? Did it use required tools when the task genuinely required them? This is closer to testing a competent engineer than snapshot-testing a string.

Large variance across runs signals that behavior is unstable and needs clearer instructions, stronger graders, or more repetitions. A plugin that sometimes works brilliantly and sometimes fails is less valuable than one that consistently delivers solid results.

What This Means for Plugin Developers and Teams?

Claude Code plugins package Skills, agents, hooks, MCP servers, LSP integrations, and other reusable behaviors. Anthropic positions plugins as the shareable, versioned format for distributing Claude Code customizations across projects and teams. The new eval command adds a quality gate that traditional software development has long relied on.

Traditional software follows a familiar progression: code flows through unit tests, integration tests, regression tests, continuous integration pipelines, and then release. Agentic software, which uses AI agents to perform tasks autonomously, needs a similar layer. The difference is that LLM behavior is probabilistic. A serious eval suite therefore measures both quality and stability before a plugin reaches production.

For developers working with proprietary code or prompts, Anthropic recommends verifying the options in your installed build before running large suites. The current build-inspection references describe local JSON and HTML reports, with a --no-publish flag available to keep reports local rather than sending them to external services.

This release represents a maturation of AI-assisted development. As Claude Code plugins become more sophisticated and more widely deployed, the ability to measure their actual contribution rather than assume it becomes essential. The claude plugin eval command transforms plugin development from art into engineering, where improvements are measurable, regressions are caught, and value is quantified.