CLI agents at Microsoft: 24% more merged PRs
+24% is for early active adopters. Who adopts the agents, and how to read the tool gap
Slide contents
1. CLI agents at Microsoft: 24% more merged PRs
+24% is for early active adopters. Who adopts the agents, and how to read the tool gap
2. Two studies answer five questions
Adoption covers Copilot CLI only; outcomes cover both tools
3. Agents cost millions; output must be measurable
Per Fortune, quoted in the paper: one Meta user in 30 days, over $1.4M
4. The window ends before Claude Code's withdrawal
IDE Copilot had been there for years; the study is about terminal agents
5. The sample drops everyone with Claude Code
And anyone who ran Copilot CLI before January 5, 2026
6. Retention: five of the first fourteen days
First use is modelled week by week; retention once per adopter
7. Engineers try more where colleagues already do
Odds of first use vs. an engineer with no colleague users
8. Peer influence is not separated from similarity
Association, not proven influence: similar engineers cluster
9. IDE Copilot users try more, stay less
Days of IDE Copilot in the 13 pre-rollout weeks vs. none
10. Engineers who ship more PRs stay
Mean weekly PRs before rollout vs. none
11. Level and tenure barely matter
Only modest shifts are significant, nearly all in first use
12. Output: a PR merged within 28 days
Azure DevOps only, active engineers only, tails beyond P5–P95 trimmed
13. Ten non-user groups form the control
CausalImpact estimates what the cohort would have merged without agents
14. Early adopters merged 24% more PRs
Week-one cohort, active engineers only; interval +14.5% to +33.7%
15. The lift held for four months
February +29.4%, March–April +20.0%: intervals overlap
16. More agent days, more PRs
Compared with the engineer's own zero-tool weeks: association, not cause
17. Copilot CLI: a 2.2× larger lift
Claude Code +11.4% vs. Copilot CLI +24.9%, single-tool users only
18. The tool gap is not yet explained
Two author hypotheses and two caveats on the comparison itself
19. IC2 and senior managers show larger lifts
Lift at three days a week; references IC4 (+21%) and 5–15 years' tenure (+22%)
20. Selection checked; quality and transfer remain open
One company, early 2026, Azure DevOps only, 16 weeks
21. Output, time and understanding are different measures
Microsoft counts PRs, METR task time, Anthropic code understanding
22. Your rollout needs controls and quality measures
What to borrow from the paper's design, and what it lacks
23. Agents move PRs but don't prove value
First use tracks colleagues
Heavy coders are the ones who stay
Early active adopters: +24% PRs, no fade
The tool gap is still unexplained
Quality, cost and delivery stay open
Measure a rollout as an experiment, not a survey