Early-2025 AI and Experienced Developer Productivity
Episode participants
What we discussed on the recording
Alexander Polomodov and Artem Aryutkin examine METR’s experiment on AI and experienced open-source developers. Participants solved real issues in familiar repositories with and without Cursor. The result showed a significant slowdown, although developers expected acceleration and still perceived the tool as useful.
The discussion inspects the design before the headline: funding, recruitment, comparison groups, and unreliable time estimates. Participants knew their codebases, but some had little Cursor experience. A mature repository full of implicit context was among the hardest settings for the assistant.
A root cause may sit far from its symptom, while crucial knowledge exists only in a contributor’s head. Treat the tool like a new teammate: ask it to read and summarize the architecture, then assign small tasks and increase difficulty after verification. This calibrates trust and exposes extra review.
The experiment cannot prove that AI always slows development: tools, user skill, task type, and sample are constrained. One task may take longer while parallel agents increase throughput. A stronger follow-up needs trained users, current models, quality measures, screen recordings, and observable outcomes.