A new study of Microsoft's early 2026 rollout of command-line AI coding agents found that developers using Claude Code and GitHub Copilot CLI merged roughly 24% more pull requests over four months, but the headline number masks critical questions about code quality, review burden, and actual productivity gains.

What Does a 24% Increase in Merged PRs Actually Mean?

Researchers Emerson Murphy-Hill, Jenna Butler, and Alexandra Savelieva published their findings on arXiv in early July 2026, studying Microsoft's rollout of two command-line AI agents: Anthropic's Claude Code and GitHub Copilot CLI. The 24% lift in merged pull requests persisted across the four-month observation window, making it one of the most concrete measurements in the often-anecdotal world of AI coding tools.

However, the researchers were careful to distinguish between output volume and actual value. Merged pull requests do not automatically translate to shipped product value. A higher PR count can mask smaller contributions, increased review workload, code rework, churn, and whether the merged code actually solved the right problem. The study treats merged PRs as a useful output metric with visible boundaries, not as a complete return-on-investment model.

How Should Teams Actually Measure AI Coding Agent Success?

The study's authors recommend a disciplined approach to measuring CLI agent impact. Rather than relying on a single productivity number, teams should track multiple dimensions of tool adoption and output quality. This approach mirrors existing frameworks like the AI Change Risk Matrix and the AI Verification Ladder, which match tools to specific work types before measuring results.

• Initial Use vs. Sustained Retention: Adoption spread through visible peer use, with reviewers, skip-level peers, and direct managers serving as signals that another engineer might try the tool. However, sustained use correlated more with an engineer's coding activity level than with demographics or job title, suggesting that rollout plans built only around training sessions or blanket enablement miss the point.
• PR Throughput Plus Review Burden: More merged pull requests are not free if reviewers are absorbing higher uncertainty, larger diffs, or lower-context changes. Teams should track whether review load increases proportionally to PR volume.
• Rework and Rollback Rates: If agents increase output but also increase revert rates, bug fixes, or follow-up cleanup work, throughput alone will misrepresent true productivity. Teams must separate initial merges from downstream corrections.
• Cost Per Accepted Change: Command-line agents can consume tokens rapidly. Organizations should track cost by task class, not just total spending, to understand where the tool delivers value.
• Task-Specific Impact: Separate scaffolding, test generation, codebase search, migration work, review preparation, and incident fixes. One blended productivity number is too blunt to guide rollout decisions.

The researchers emphasized that adoption mechanics matter as much as raw output numbers. The best early candidates for CLI agent adoption are not necessarily the loudest AI enthusiasts, but rather people with enough active coding work for the tool to become part of their daily loop.

Steps to Plan a Responsible CLI Agent Rollout

• Define the Eligible Population: Start with active contributors in real repositories rather than rolling out broadly. This creates a measurable baseline and makes successful workflows visible to reviewers and neighboring teams.
• Separate Trial From Retention: Track initial use and sustained use as distinct metrics. A first-week launch spike tells you about curiosity; sustained use tells you whether the tool found a real workflow fit.
• Track Output Against a Baseline: Measure merged PRs, code quality signals, and review burden before and after adoption to isolate the agent's impact from other variables.
• Add Quality and Cost Measures: Include rework rates, rollback frequency, token consumption, and cost per accepted change. These prevent throughput metrics from hiding downstream problems.
• Decide on Broad Access: Only after gathering data should teams decide whether broad access is a productivity investment, a training investment, or simply an expensive way to make the PR queue busier.

Why Adoption Patterns Matter More Than You Might Think

The study's adoption findings may prove more actionable than the 24% productivity number itself. Initial use spread through visible peer use, meaning engineers trust tools faster when the person reviewing their PR, pairing with them, or managing their team visibly uses the same tool in real work. This fits the lived reality of developer tools and suggests that rollout strategy should prioritize visibility and social proof over blanket enablement.

Retention, however, looked different. Sustained use was associated more with the engineer's coding activity than with demographics or seniority. This means a rollout plan built only around training sessions or job level is likely to miss the point. The practical implication is clear: start with active contributors in real repositories, make successful workflows visible to their reviewers and neighboring teams, then measure whether use survives beyond the first two weeks.

The observation window for the study ended on April 29, 2026, before an internal shift away from most Claude Code licenses would have distorted the results. This timing allowed the researchers to capture genuine adoption patterns without the noise of organizational changes.

For teams planning their own CLI agent rollouts, the takeaway is not "expect 24% everywhere." Instead, the message is clearer and more practical: build an instrumentation plan before you buy a broad license. Copy the study's shape before copying its number. Define the eligible population, separate trial from retention, track output against a baseline, add quality and cost measures, and then decide whether broad access makes sense for your organization.