[1/2] Claude Code and Expertise: What They Show 398 thousand (Category AI4SDLC)

[1/2] Claude Code and expertiseWhat they showed 398 thousand (Rubric #AI4SDLC)

Added to the body AI4SDLC meta-study 2026 fresh-work (16 June 2026) Anthropic "Agentic coding and persistent returns to expertise“, the title of which concludes that the coding agents do not cancel the examination. But what's more interesting to me is the authors' approach to measuring the division of labor between humans and Claude Code in real-world work sessions. This is a large but not experimental source. Anthropic took a random sample from 398 198 interactive sessions 234 751 user from October 2025 April 2026 years. The sample included Claude Code in CLI, Claude.ai and desktop application. Internal sessions of Anthropic employees, third-party IDE, SDK and headless launches were excluded. This is a strong observational data of a single product, not an independent study of the entire software development.

The method is interesting. Model classifiers read transcripts with privacy and marked:

• what kind of work was performed; Who made the planning and execution decisions;
• what subject competence the user showed;
• how the session ended.

The authors compared part of the classification with independent telemetry: the number of references to the model and tools, the output volume, added and deleted lines. A commits, tests, and user confirmation has been looking for a separate transcript success classifier. The researchers did not read individual private transcripts, but published aggregates.

The authors understood the competence of users in the current task. (not grade, seniority or title). To assess the expertise, the authors used a competency assessment classifier as part of a five-level task: how accurately it defines the framework, understands the terminology, asks for specific risks to be checked, and corrects Claude substantively. A senior developer writing in Rust for the first time may be a beginner in Rust coding. An accountant with no Python experience who pinpoints the reconciliation rules and catches an error in handling the closing of the month is an expert in the accounting task. The study does not compare junior and senior engineers.

The results were interesting.

1The division of labor is already noticeable In a typical session, a person took 70% of decisions about what to do, but only 20Percentage of decisions on how to do it. In other words, the user basically held the goal and criteria, and Claude was the choice of files, commands, and implementation details. This fits well with the chain I wrote about in the State of AI4SDLC: intent → context → plan → tasks → implementation → verification. Anthropic's analysis doesn't prove that the entire SDLC has already rebuilt, but it does show a similar pattern within interactive agent work.

2Different levels of competence led to different levels delegation In sessions labeled as novice, one request ran approximately 5 actions of Claude and 600 In addition, in expert sessions, about 12 action 3 200 words. The link was maintained after taking into account work mode, task value, month, profession and model family. But “more action and text” doesn’t mean “more useful work.” A long chain can be an autonomous execution, or perhaps unnecessary iterations and alterations. As long as this is a measure of delegation, not productivity. ** 3Higher assessment of expertise was associated with more frequentverified success\** After the statistical adjustment, this indicator increased with 14,5% up to novice 20,9% at beginner, 28,3% at intermediate 29,5% advanced and 32,9Percent expert. The main jump occurs between novice and intermediate; between intermediate and expert, the increase is noticeably smaller. Verified success here means that the classifier found the goal achieved and found at least one signal: passed the test or command, suitable commit/PR or explicit user confirmation. This is stricter than a simple model verdict, but it does not mean that the change has reached production and benefited.

I already am. handler Anthropic Agentic Coding Trends and Transition from Code Writing to Agent Management The new work adds to this line an observable pattern: the agent takes away most of the execution, and the task-specific expertise is associated with more frequent session success.

And next I will examine the unpleasant question: how reliably expertise and success are measured here, why correlation is not yet causality, and what a single randomized experiment says about skill formation.

#AI #AI4SDLC #Research #Engineering #Agents #Metrics #DevEx