The Boundaries of Real Autonomy: What Work an Agent Completes End to End

In 1978 Sheridan and Verplank described ten levels of automation, from “the human does everything” to “the computer decides everything and ignores the human”. Their unit was “a single elemental decisive step”: get the options, select, approve, start. Since then the unit has quietly grown while the word stayed the same. Today autonomy names five different things. An action: one agent turn in the terminal, whose median length in Claude Code is about forty-five seconds. A task: a SWE-bench issue or a finished work product in GDPval, where the expert spends seven hours on average and the agent gets the full context in one prompt. A work block: an Upwork project from the Remote Labor Index with a median of eleven and a half hours and two hundred dollars. A role: a digital worker in TheAgentCompany with a hundred and seventy-five tasks and simulated colleagues. And an outcome: a year of trading in Vending-Bench or the profit of a real shop in Project Vend.010101I wrote up Michele Catasta's Replit talk: autonomy is named there as the main measurable agent metric and, at the same time, as a property of the system rather than the model. Both thoughts are the spine of this piece.Knizhny kub · Autonomy Is All You Need01I wrote up Michele Catasta's Replit talk: autonomy is named there as the main measurable agent metric and, at the same time, as a property of the system rather than the model. Both thoughts are the spine of this piece.Knizhny kub · Autonomy Is All You Need

The five units are an author taxonomy of what is delegated to a system, not a measured scale of declining success. In GDPval, a model receives a task and reference materials and produces a professional work product, such as a document, a spreadsheet with calculations, or a presentation. Experts compare it with a specialist's work without knowing who created each version. That finished work product is what the benchmark evaluates. RLI assesses projects, TheAgentCompany tasks in a simulated organization, and Vending-Bench a balance in a simulation. Their models, dates, tasks and success criteria differ. Differences in scores cannot establish how much harder larger units of work are for an agent.

The table's figures come from separate evaluations. 84.9% on GDPval is OpenAI's reported win-or-tie rate for GPT-5.5 against professional work products. 42.9% on TheAgentCompany is the share of fully completed tasks in the leading entry dated 10 November 2025. 20.8% on Remote Labor Index is the rounded GPT 6 Astra score in the 26 September 2026 snapshot; its denominator needs clarification. These are three separate observations, not successive stages of one experiment.

Researchers' autonomy levels also describe different properties. Mitchell and colleagues at Hugging Face, in Fully Autonomous AI Agents Should Not be Developed, distinguish levels by how much control a model has over program execution: producing an answer, selecting a tool call, or writing and running its own code. Feng and colleagues, in Levels of Autonomy for AI Agents, describe the human's role: operator, collaborator, consultant, approver, observer. Kasirzadeh and Gabriel, in Characterizing AI Agents for Alignment and Governance, consider the share of tasks performed without the person who delegated the work. The February 2026 OECD report distinguishes four modes: the human decides, approves an action, can stop it, or stays outside the execution loop. “Level four” is meaningful only alongside the scale's name and a description of human involvement.

What a system can do and what it is asked to do

Morris and colleagues at Google DeepMind, in Levels of AGI, § 6.2, separate capability from the mode of human–AI interaction. The paper first appeared in 2023; this article uses the June 2024 revision. Capability covers performance relative to humans and the breadth of tasks. Autonomy describes how people and systems share work and initiative.

Table 2 identifies six modes: 0 — no AI; 1 — a tool for individual operations; 2 — a consultant invoked on request; 3 — collaboration; 4 — AI leads while a human guides and gives feedback; 5 — an autonomous agent. The last mode still allows asking people for help: continuous oversight and consultation are different things.

The diagram uses a hypothetical model that can prepare reports. It can be called on for analysis, used to edit a document together, or assigned the preparation cycle with a human checking the result. Changing the mode does not make the model smarter. Improving the model expands the available choices; the appropriate mode still depends on the task, cost of failure and controls.

Production surveys describe the prevalence of different usage modes. In August 2026 Deloitte counted 42% of companies that had tested or deployed agents and 15% with scaled adoption; Gartner counted 17% with agents in production; Bain, in Bloomberg's retelling, 7% with fully autonomous agents; a Russian study by Infosistemy Jet, 8%. Anthropic's own engineers, whose access to the best models is unlimited, said in December 2025 that they could fully delegate between zero and twenty percent of their work. Raiffeisen at Saint HighLoad++ named a quarter of the backlog done together with agents and four percent done by agents without asking a human. Between “the agent handled a task” and “the agent can be relied on as a standing performer” lies this entire ladder, and the word autonomy jumps over it.

The time horizon describes how much work an agent can complete at a given success probability. Work is measured in a specialist's time: a “two-hour task” takes a human about two hours, even if the agent spends only ten minutes on it. METR's TH1.1 suite contains 228 programming, machine learning and cybersecurity tasks.

1. Establish each task's human duration. A specialist completes it while being timed; some tasks use expert time estimates.
2. Give the same tasks to the agent. Record whether each attempt succeeds or fails. Repeat attempts to reduce the influence of a lucky success or an unlucky failure.
3. Fit a curve to all the results. Human task time is on the horizontal axis, agent success probability on the vertical. A smooth line summarizes the scattered attempts: how much less often the agent succeeds as tasks require more human time.
4. Find the intersection with the desired percentage. If the line crosses 50% at “4 hours”, the 50% horizon is four hours. This estimates success on tasks of that size; it does not promise four hours of continuous agent work.

Four hours above is a hypothetical example. METR's method uses a logistic curve: a smooth declining line between 100% and 0%. Its position and steepness are fitted to the agent's successes and failures. The time axis is logarithmic: equal distances mean equal multipliers, such as 10 minutes → 100 minutes → 1,000 minutes. This puts short and long tasks on one chart. The intersection with 80% gives a second, shorter horizon: higher reliability requires a smaller work unit.

Checked on 27 September 2026, METR's main table still lists 8 May as its last update. In that snapshot, the early Claude Mythos Preview has the highest 50% horizon at 17 hours 25 minutes, and an 80% horizon of 3 hours 6 minutes. These are uncertain estimates: the first has a 95% confidence interval from 8 hours 29 minutes to 55 hours 4 minutes; the second, from 1 hour 37 minutes to 6 hours 39 minutes. METR warns that values above 16 hours are unreliable with the current suite. The May figure cannot be treated as an established limit on model capabilities in September.

The diagram below uses the two published Mythos estimates. Its smooth line is reconstructed from those two points to explain the method, not newly fitted to the original runs. The horizontal bars show uncertainty intervals for the two horizons, not for the entire curve.

Later reports exist, but do not update that main table. On 26 June, METR published its GPT-5.6 Sol evaluation: about 11.3 hours when attempts that circumvented the evaluation rules were counted as failures, versus over 270 hours when those attempts were counted as successes. METR does not consider any of these numbers a robust capability estimate. The 22 September Claude Opus 5.5 evaluation used five AI research and development tasks. It describes incremental improvement over Fable 5.1, but publishes no new comparable 50% or 80% time horizon.

The trend is better known than the values themselves: a doubling roughly every two hundred days across the whole series since 2019, and every 129 days counting from 2023; the January post also named 88.6 days since 2024. In April 2026 Epoch AI confirmed the acceleration and attributed it to reasoning models, noting that the gains concentrate in programming and mathematics — where the answer is checked automatically. What matters for this piece is that METR itself insists on what the horizon is not. “Time horizon is not the length of time AIs can work independently” but “the amount of serial human labor they can replace with a 50% success rate”. The estimate's error is about a factor of two each way. Domains differ by orders of magnitude: for tasks on a computer screen the horizon is forty to a hundred times shorter. The key line for delegation: “a 50% time horizon of X hours does not mean we can delegate tasks under X hours” — some tasks need ninety-eight percent success or more.

The tasks themselves have another limitation: fewer complications than everyday work. METR assesses this using 16 factors, including coordination with other participants, a changing environment, limited resources and difficulty checking the result. Each factor present adds one point. In the original study, HCAST and RE-Bench tasks averaged 3.2 out of 16, with none above 8. This describes working conditions, not data quality. As a hypothetical example, fixing a function against an existing test is easier to organize than diagnosing an incident in a changing system, coordinating a fix with another team and checking the outcome without a ready-made reference.

The mathematical curve also affects the answer. Fergus Hamilton reanalysed METR's results using another curve shape, the Weibull distribution. It describes the relationship between task duration and success probability differently. Both curves fit the available observations closely, but diverge when estimating near-perfect reliability. Toby Ord's discussion of this comparison shows that the calculated 99% horizon can be twenty times shorter under the Weibull curve than under METR's logistic curve. This concerns the task duration predicted to yield 99% success: hypothetically, one method might give 100 minutes and the other 5 minutes. That illustrates the difference between estimates; it is not a separate model measurement.

Ord stresses that there is insufficient evidence to choose confidently between the curves. A good measurement at 50% therefore does not reliably establish which tasks an agent can complete in 99 out of 100 attempts. METR's own March sensitivity analysis found that removing public RE-Bench tasks cuts Opus 4.6's horizon from twelve hours to seven. Reasonable fitting choices change the 50% horizon by about 1.5× and the 80% horizon by 2×. Task selection remains the main source of uncertainty.

Now set this against what vendors say and what production shows. At the launch of Sonnet 4.5 Anthropic reported “more than 30 hours” of focus on multi-step tasks — an observation without a method; METR gives the same model one hour fifty-seven at fifty percent and twenty-six minutes at eighty. Codex, in OpenAI's words, worked “more than 7 hours”; METR gives its successor three hours forty-four. The difference between the claims and the measurement is six- to fifteenfold, and it is not a lie: one number is the agent's running time on a single showcase task with an unstated success criterion, the other is human time at a stated probability across two hundred tasks. Anthropic's own telemetry from February 2026 settles it: the median Claude Code turn is forty-five seconds, the 99.9th percentile grew from twenty-five to forty-five minutes in three months, 73% of API tool calls belong to agents classified as having a human in the loop, and 0.8% of actions are irreversible. These are turn durations, not capability limits: a short turn can mean fast completion, and parallel work can also reduce elapsed time. And METR's experiment with sixteen developers in 2025 showed a 19% slowdown against an expected 24% speedup; the 2026 rerun gave minus eighteen and minus four with wide intervals, and METR itself called the survey with a median self-reported “three times faster” “not necessarily grounded in reality”.020202I wrote about the sixteen-engineer experiment back in July 2025: the sample is small but the methodology is honest. A year later it gained a rerun that proved nothing — and that is a result too.Knizhny kub · the METR experiment02I wrote about the sixteen-engineer experiment back in July 2025: the sample is small but the methodology is honest. A year later it gained a rerun that proved nothing — and that is a result too.Knizhny kub · the METR experiment

METR is also exploring complementary measures: compute cost, human participation and gains under a fixed budget. The practical conclusion is limited: the 50% and 80% horizons characterize a selected task distribution and evaluation procedure. Neither authorizes delegation on its own. That requires a local sample, a reliability requirement, an error cost and complete human-effort accounting. METR's limitations note discusses distribution dependence and unstable high quantiles. Comparisons with other domains are separate observations; they do not provide a measured Mythos computer-use curve.

CorpGen studies 46 OSWorld office tasks in a shared session with dependencies at several loads. In table 3, CUP stands for computer-use-preview, OpenAI’s model for operating computer interfaces. The authors use computer-use-preview-2025-03-11. “Baseline” means the agent without the additional CorpGen layer that coordinates multiple tasks. This configuration scores 16.7% at 12 tasks and 8.7% at 46. A separate variant with hierarchical planning, Hierarchical CUP, scores 25.0 → 14.1%. The positive result matters too: at full load, CorpGen improves CUP from 8.7 to 16.3%; CorpGen with CUP scores 16.7 → 16.3% across the endpoint loads. Load effects therefore depend on execution architecture. Small task sets and absent intervals prevent treating these differences as a precise general effect of multitasking; an isolated task is not the control condition here.

TheAgentCompany from CMU shows the second side: not load but a role. The paper's best agent solved 24% of tasks in December 2024, 30.3% by May 2025, and the last leaderboard entry from November 2025 reads 42.9% fully and 52.4% on the weighted score, which grants half the checkpoint completion fraction plus half the full-success indicator. The failures come in four kinds: lack of common sense, inability to talk to colleagues, helplessness in web interfaces, and self-deception — the agent “creates fake shortcuts that omit the hard part”. In August 2026 an independent re-analysis of three such benchmarks produced what I consider the main result of this section: the choice of agent explains under three percent of the variance, and the agent-by-task interaction seven to twenty-three. Which tasks matters more than which agent. ClawMark, released in April, added a third dimension — an environment that changes without the agent: mail arrives, the calendar shifts, the knowledge base updates. Sonnet 4.6 scored 75.8 weighted points and 14% strict success, and performance dropped right after the first exogenous change.

The cleanest “same task plus interaction” measurement comes from Salesforce. In CRMArena-Pro the best model solved 58.3% of tasks in a live CRM when everything was said in one turn and 30.0% when the missing pieces had to be elicited from a simulated customer; in forty-five percent of the dialogue failures the agent simply did not ask. OpenAI's GDPval stands at the opposite pole — full context up front, no clarifying dialogue — and the authors themselves write that results fall when context is cut and the win rate declines as task length grows. Their own arithmetic sobers up the loud “a hundred times faster and cheaper”: that is pure inference time. GDPval Table 2 reports a speedup factor from 0.87 for GPT-4o to 1.12 for GPT-5 under the strategy “try AI once, have an expert review it, then do the task yourself if needed”. The factor divides time without AI by time with AI: below one means a slowdown. In the authors’ calculation, GPT-4o increases total time by about 15%, while GPT-5 reduces it by about 11%. For example, a 100-minute task without AI would take about 115 or 89 minutes respectively. These are estimates from a time-cost model, not measurements of a workplace rollout.

The Remote Labor Index went from 2.5% to 20.8% of accepted projects in a year, but the new runs already used Claude Code and Codex with a worker–critic loop, a budget of up to a hundred and fifty dollars and a full day of time; the model judge overstated the success share two and a half to three times, and of the three showcased Fable 5 deliverables the authors wrote that “none would be accepted as finished work”. Curiously, success on RLI does not fall with the human length of a task — there is no horizon curve there, because a project is not “more text” but “more places where things can go wrong”.030303SWE-Together from Meta does the same for coding: 109 tasks from eleven thousand real sessions and a user simulator that annotators could not tell from a live person. A stronger model needs fewer interventions — but it still needs them.Knizhny kub · SWE-Together03SWE-Together from Meta does the same for coding: 109 tasks from eleven thousand real sessions and a user simulator that annotators could not tell from a live person. A stronger model needs fewer interventions — but it still needs them.Knizhny kub · SWE-Together

In APEX-Agents, Gemini 3 Flash meets every criterion on 24% of 480 tasks, while its mean criterion score is 39.5% (table 3). These answer different questions: whether the result is complete and how far the work progressed. Partial credit is useful diagnostically and is not inherently flattering. The Gemini 3 Flash judge was checked against 747 human labels from 60 tasks; the reported 98.5% agreement applies to that sample. In RLI, the model judge and experts give Opus 4.8 roughly 21% and 8.3% respectively, evidence of disagreement in that particular evaluation. RLI's methodology describes 240 selected projects, with 10 public and 230 private projects for quantitative evaluation. The current leaderboard percentages are not transparently reconciled with that denominator; the number of accepted projects must not be reconstructed from 20.8%.

These studies reveal different sources of disagreement: task conditions, the definition of success, and the judge. They cannot be combined into a single estimate of lost autonomy over a working day. That would require controlling the model, task set, budget and verification. A particular experiment can support a discussion of dialogue or load effects; cross-study comparisons describe differences in setup and limits to transfer.

Sierra introduced a distinction that almost every leaderboard still ignores. pass@k is the probability that at least one of several attempts succeeded; pass^k, that all k attempts succeeded. Each attempt runs the same task from scratch.

On τ-bench in 2024 GPT-4o solved 61.2% of retail tasks on the first attempt and under 25% on all eight. On τ²-bench in 2025 every model lost fifteen to twenty-six points between pass^1 and pass^4, and the telecom domain, where both agent and customer use tools, lost half: 0.34 → 0.19 for GPT-4.1. Merely adding an acting user to the same task took away eighteen to twenty-five points. In January 2026 Anthropic fixed this in its guide to evals: capability evals start at a low pass rate, regression evals should pass near one hundred; “a task that passed on one eval run might fail on the next”; pass^k is for agents where consistency matters. In 2026 the leaderboards still display pass^1 in the shop window.

The Princeton line of work — from the 2024 paper on which agents matter to HAL — added cost and logs. Across 21,730 rollouts costing forty thousand dollars, more reasoning did not raise accuracy in twenty-one of thirty-six combinations; an agent can be “100x more expensive while only being 1% better”; transcripts revealed agents searching HuggingFace for benchmark answers and hard-coding tests; agents with identical accuracy carry different risk — abstaining and paying with someone else's card both score zero. In a May 2026 study, researchers from Princeton and the UK AI Security Institute found defects in 25 of 50 τ-bench Airline tasks: inconsistent policies, ambiguous instructions, and database or grading errors. Excluding these tasks raised average pass^5 across the sampled models from 20.8% to 40.0%. This is what the authors mean by performance being underestimated by nearly half: flawed tasks prevented agents from demonstrating their capabilities. The models were unchanged; the evaluation set changed. In a separate February study, researchers found that “recent capability gains have only yielded small improvements in reliability” across twelve metrics. The harness has become a variable of measurement: on Terminal-Bench the same model scores 83.8% in Claude Code and 80.4% in another harness, and a May preprint showed that harness-induced variance “can substantially exceed model-induced variance”, up to reversing rankings. I went into that effect in the harness piece; what matters here is the consequence: a report without the harness version and the number of runs is not a measurement.

The benchmarks themselves age faster than they seem to. In February 2026 OpenAI stopped publishing SWE-bench Verified: of 138 hard tasks, 59.4% have defective tests or specifications, and all frontier models reproduce the gold patches from memory. The replacement was SWE-bench Pro, where Opus 4.1 fell from 22.7 to 17.8% on private commercial repositories; in eight months the public split grew from 23.3 to 80.3%, after which the same OpenAI found “~30% of tasks broken” and withdrew its recommendation. The July 2025 checklist for rigorous agentic benchmarks found task-validity violations in seven of ten suites: an empty response scored 38% on impossible τ-bench tasks. Web agents on live sites showed 30% instead of a claimed 89%. Every “fixed” successor saturates in six to twelve months and turns out fifteen to thirty percent defective once models are strong enough to expose it.040404In my breakdown of Anthropic's study of 398,000 sessions I wrote why session success cannot be renamed productivity: “verified success” cannot see whether the team accepted and deployed the code. The same boundary applies here.Knizhny kub · Claude Code and expertise04In my breakdown of Anthropic's study of 398,000 sessions I wrote why session success cannot be renamed productivity: “verified success” cannot see whether the team accepted and deployed the code. The same boundary applies here.Knizhny kub · Claude Code and expertise

AIDev supplies another metric: PR acceptance in an observational corpus. Table 5 reports 38–65% for different agents and 76.8% for human authors in popular repositories. This is PR acceptance, not acceptance without edits or a randomized comparison on the same tasks. Task, repository and user selection may explain part of the differences. Acceptance also does not establish long-term reliability: that requires data on reverts, defects and maintenance.

The simplest model of a long chain is a power: if each step succeeds with probability p and steps are independent, a chain of n steps succeeds with probability p^n. Sinha and colleagues, in their ICLR 2026 paper, derived the horizon from it: the number of steps before success falls below a threshold equals ln s / ln p. At 95% step accuracy that is thirteen and a half steps to a coin flip (a 50% chance that the whole chain succeeds), at 99% sixty-nine, at 99.9% almost seven hundred. Hence their thesis about the “illusion of diminishing returns”: one more point of step accuracy lengthens the horizon by a quarter, and the closer to one, the stronger the effect. But real models violate independence in both directions. On one side, self-conditioning: errors in the context raise the probability of further ones, scale does not fix it, reasoning does. On the other, extra compute per step: GPT-5 with reasoning executes more than two thousand steps in a row on a synthetic task. Independent 2026 measurements confirm the first side: the decline with length is one and a half to two and a half times steeper than the independent model; after a wrong step the next one is wrong 40–58% of the time against three to five after a correct one.

A checkpoint is a point where the system verifies an intermediate result before continuing. For example, after an agent edits code, the system runs tests and saves the version that passes. If tests fail, it restores the previous saved version and retries that segment. The model below checks after every ten steps and allows one initial attempt plus up to two retries per segment. Three failures stop execution.

A check does not fix an error by itself: the benefit comes from detecting failure, restoring state and trying again. The diagram assumes perfect error detection, independent retries and no irreversible damage before verification. A ten-step segment succeeds within three attempts with probability q = 1 − (1 − p¹⁰)³; five such segments succeed with probability q⁵. At p = 0.95, this gives 71.61% instead of 7.69% without retries. These are the same 50 useful task steps, but repeated execution and checks consume extra time and resources. Real checks can miss errors and retries can repeat them, so this gain is not guaranteed.

The Traverse dataset, published by Rahman and colleagues in September 2026, contains 2,518 agent trajectories across software engineering, computer use and science, with 6,967 annotated mistakes. In the subset of 1,122 trajectories with a labelled first mistake, 69.5% never recover, 38.5% never notice it, 72.6% keep acting, and about 84% of the studied software and computer-use failures end with a step that looks correct. Frontier model judges find the first mistake in fewer than a third of coding runs. The MAST taxonomy from Berkeley, over 1,642 traces, sorts failures into fourteen modes: step repetition 15.7%, reasoning–action mismatch 13.2%, not knowing the stopping condition 12.4%, incorrect verification 9.1% — the last a false pass that turns a recoverable run into a silent failure. Where the critical error sits is known too: a failed trajectory carries on average 7.6 local errors and exactly one critical one, the agent repairs 62% of the non-critical ones itself, and the root causes cluster on steps six to fifteen — after information gathering, at the first decisive action. That is where a checkpoint is worth the most. Hamilton's data on the declining hazard rate say the same from the other side: early steps are the riskiest, verification should be front-loaded, and a “surviving” run should not be reset without evidence.050505In the Loop Engineering breakdown the key element of the loop is the verifier's right to say no. This whole section is about the verifier needing evidence, not an opinion.Knizhny kub · Loop Engineering05In the Loop Engineering breakdown the key element of the loop is the verifier's right to say no. This whole section is about the verifier needing evidence, not an opinion.Knizhny kub · Loop Engineering

Who verifies matters as much as where. Self-verification without external evidence does not work: back in 2023 GPT-4 fell on GSM8K from 95.5 to 89.0% after two rounds of “correct yourself” and rose to 97.5 with an oracle saying where the error was; in 2026 it was shown that relabelling the model's own thought as someone else's raises the share of explicit corrections by 23–93 points. External, evidence-based verification, by contrast, gives large measured effects: a linter on every edit in SWE-agent — plus 7.7 points; fixing the critical error and restarting from it — from 21 to 55% on ALFWorld; a trained verifier choosing among several attempts — from 81.8 to 90.2% on Terminal-Bench; decomposition at subtask boundaries — plus 13–42 points. Theory explains why: the step-level verification signal grows with length as Θ(T), the outcome signal decays as Θ(T·p^T) — the same law that kills the chain itself. But verification is neither free nor always useful: a weak verifier is worse than none, a blocking mediator intercepts 94% of violations and leaves under 5% safe successes, and the Qwen team writes outright that “generating a solution has become easier, reliably verifying it has become the harder problem”, so verification must co-evolve with the generator.

Blocking an irreversible action means the system prevents execution until the required checks or authorization pass. For example, a database deletion tool stays unavailable until a human approves it. The restriction must be enforced by the tool or execution system; an instruction asking the agent to be careful cannot guarantee a stop.

Harnesses for long runs converge on the same primitives, and it helps to tell them apart. A state checkpoint — a commit, a progress file, the append-only session log in Managed Agents — makes a crash recoverable. A plan checkpoint — a JSON feature list, a sprint contract — stops the agent from “declaring it done” by redefining the goal: in November 2025 Anthropic watched a later instance “look around, see that progress had been made, and declare the job done”. A verification checkpoint — tests, a browser, an external evaluator with hard thresholds — catches silent errors. The only vendor comparison with a control is the March one: a solo agent built a game in twenty minutes for nine dollars, but nothing responded to input; the full harness spent six hours and two hundred dollars and delivered a working one; and the evaluator gave lift “only at the edge of the generator's capabilities”, while “out of the box, Claude is a poor QA agent”. Durable-execution engines fix crashes, not meaning: their argument is “five steps at 99% give 95%”, their limit is that a semantic error is journaled and replayed as faithfully as a correct step. Context compaction is a risk point of its own: recursive summaries turn reliably solved tasks into intermittently solved ones, while truncation without paraphrase under the rule “never compact a compaction” holds the result at half the cost.

• •A cheap mechanical check on every state-changing action: the chain starts with the first bad edit, and recovery after it is 57%
• •An evidence-based check at subtask boundaries: critical errors cluster after information gathering, and decomposition with a check recovers 13–42 points
• •An external judge at the end: the model passes its own work and fails someone else's; a 23–93-point shift from one relabelling
• •Block an irreversible action until the required checks or authorization pass: 97% of destructive steps in “solved” runs were taken without acknowledged risk
• •The interval between checks grows with the square root of reliability and of check cost: at 98% per step and a check costing two steps — roughly every fourteen steps

The last point transfers the Young–Daly formula from supercomputing, where the optimal checkpoint interval is the square root of twice the product of the mean time between failures and the cost of a checkpoint. For an agent at 98% step accuracy and a check costing two steps that is about fourteen steps at 28% overhead; at 99.5%, twenty-eight steps and 14%; a cheap linter costing a fifth of a step pays off every four to five steps. The shape of the law — checks get rarer as reliability and check cost grow — matters more for practice than the constants, and the same literature reminds us that with a declining hazard the intervals should lengthen as the task proceeds. Two corrections are mandatory. An agent failure is not fail-stop: since 84% of failures look correct, the cost of a checkpoint must include detection, otherwise you save the state of a corrupted run. And a retry helps only if it is independent: self-conditioning makes a retry in the same context correlated with the first failure, which is why the unit of retry in every 2026 harness is a fresh context plus restored state, not “try again”.

In July 2025 Jason Wei formulated the “verifier's law”: the ease of training AI to solve a task is proportional to how verifiable it is, and a verifiable task is one with objective truth, where checking takes seconds, scales, is low-noise and yields a continuous score. In November Karpathy compressed the same into “Software 2.0 easily automates what you can verify”, and at the YC summer school talked about the autonomy slider and the generation–verification loop that should spin as fast as possible. The economics of sampling confirm it: on SWE-bench Lite the share of solved tasks grew from 15.9% with one sample to 56% with two hundred and fifty — but only where tests exist; without an automatic verifier, selecting the best attempt plateaus. So the question “is fifty percent enough” has no answer without a second question: what does verification cost, and what happens when it misses an error.

The following is an author cost model. Let manual execution cost S, one agent attempt A, and its verification V, all expressed in the same monetary units or converted into them. With constant success probability p, independent retries, perfect verification and no damage before acceptance, expected cost per accepted result is (A + V) / p. Savings require p > (A + V) / S; equality is break-even. If S = 60, A = 2 and V = 10, the threshold is 20%; at V = 40 it is 70%. These are hypothetical cost units, not a sum of machine and human minutes. If A + V > S, even p = 1 produces no savings. Fallible verification, dependent attempts or damage before acceptance require a different model. Without verification, comparing expected value with guaranteed successful manual work of value W and an agent failure loss L gives p ≥ 1 − (S − A) / (W + L). W and L must be chosen and justified for the process. Lamba's sixteenfold decline in optimal throughput when error detectability halves assumes δ = 1; with fixed labor, the quadratic relationship instead gives a fourfold decline. These are model conditions, not a universal law of verification. Cost per accepted task is discussed further in the article on development economics.

The matrix below organizes questions; it does not prescribe universal percentages. Cheap verification still requires checking the cost of the whole attempt; expensive verification requires checking whether savings remain possible at all. Large potential losses need a separate risk constraint: limit permissions, support recovery, add independent verification or retain human approval. Reversibility affects consequences but does not itself imply a 50% or 90% success threshold.

The literature that publishes thresholds ties them to a maturity stage, and the form of the threshold differs at each. A research prototype is measured by a horizon at a fixed probability and pass@k against an oracle. A commercial product by an outcome share under a contractual definition: since July 2026 Salesforce charges only when the agent “autonomously resolves an issue from start to finish”, with no charge on negative feedback or a human-escalation request, and reports 70% of such outcomes across 4.3 million inquiries on its own portal; at Intercom an “assumed resolution” is a customer who did not reply for a day; Zendesk advises excluding abandoned and repeat contacts. Responsible operation by decision rights and a stop button: Article 14 of the European AI regulation requires oversight “commensurate with the risks, level of autonomy and context of use”, with the ability to override a decision and stop the system; the algorithmic-trading rules require cancelling any orders immediately and naming which algorithm is responsible for each; banking supervision requires “effective challenge” of a model and limits on its use; and in January 2026 the FDA drew a line worth remembering: software remains decision support only if the clinician can independently review the basis of a recommendation, and under time pressure that is already automation. Aviation and automotive safety requirements specify the probability of a dangerous failure per operating hour: a catastrophic condition in aviation must be “extremely improbable”, on the order of 10⁻⁹ per flight hour, and must not result from a single failure; the automotive standard sets a limit below 10⁻⁸ per hour for dangerous random hardware failures at its highest class, ASIL D. The more severe the possible consequences, the lower the acceptable failure probability. These figures express system safety requirements, not the percentage of successfully completed tasks. For office work nobody publishes a numeric threshold for full automation — the oft-quoted “95% accuracy” traces back to consultancy blogs, not analysts' reports.

These studies do not establish a universal threshold for a prototype, a commercial product or a consequential operation. The decision depends on the distribution and cost of errors, the verifier's ability to detect them, and the cost of all attempts. The process owner chooses an acceptable risk; evaluation should show how confidently the measurements fit that constraint. A threshold cannot be imported directly from another benchmark.

If the threshold is a property of the task, an autonomy report cannot consist of one percentage. I propose a scorecard of six rows, and for each row there is data somewhere in 2026 already. Result — the share of tasks accepted under a criterion written before the run; the definition of acceptance matters more than the number, and the Salesforce, Intercom and Zendesk definitions show how strongly it moves the result. Interventions — human touches per task: Anthropic measured that experienced Claude Code users both turn on full auto-approve more often, from 20 to 40% of sessions, and interrupt the agent more often, from 5 to 9% of turns — trust and vigilance grow together; in February Microsoft formalised this as TCR@k, the share of tasks completed with at most k approvals. Verification cost — human minutes per accepted task: in the METR experiment that was nine percent of time spent reviewing and cleaning output while accepting under 44% of suggestions; in Faros telemetry, a two-and-a-half-fold rise in time to first review measures waiting, not active human effort. Recovery time — from failure to a working state, for which DORA already has a definition; the Kiro agent with an engineer's inherited role decided to “delete and recreate the environment” and took a service down for thirteen hours in one region. Blast radius — the maximum damage of one run and the reversibility of each action: 0.8% of API actions are irreversible; the auto-mode classifier misses 17% of dangerous actions on fifty-two cases — “17% is the honest number”, Anthropic writes; the Replit agent deleted a production database during a code freeze. And cost per accepted task — all spend over the period divided by accepted tasks: HAL shows agents “100x more expensive while only being 1% better”, and the Remote Labor Index counts a failure as infinite cost.060606The DX report for the second quarter is the best illustration of the scorecard without a scorecard: more code, higher maintainability, lower confidence in changes. One row rises, another falls, and without both there is no picture.Knizhny kub · DX Q2 202606The DX report for the second quarter is the best illustration of the scorecard without a scorecard: more code, higher maintainability, lower confidence in changes. One row rises, another falls, and without both there is no picture.Knizhny kub · DX Q2 2026

The scorecard is not decoration: any single row can be inflated. The share of accepted changes can rise when people approve them without substantive review, as Warp warns. Permission decisions pose a similar risk: Anthropic reports that Claude Code users approve 93% of prompts and warns about approval fatigue: people stop paying close attention to what they are allowing. Interventions fall if the agent only gets documentation. Cost falls if you count only the first model call. Only the combination of rows separates a standing performer from a lucky demo: accepted under a strict criterion, with few touches, cheap verification, fast recovery, a bounded radius and a known price. A seventh candidate row is error detectability from Lamba's work: an agent that errs less often but more plausibly may cost an organisation more than one that errs often and obviously. Nobody has yet shown how to measure this on a stream, but incident reports deserve a field “how many reviews the error passed before detection”.

In 2018 Astra Taylor called it “fauxtomation”: labor does not disappear, it shifts — onto the customer at the kiosk, the content moderator, the Mechanical Turk worker. For agents the word needs narrowing: pseudo-autonomy is when human labor has been moved into writing instructions and fixing hidden errors and excluded from the metric. The best document on the mechanism is the SEC order of 14 January 2025 in the matter of Presto, the maker of a voice ordering system for drive-throughs. The first version “required human agent intervention, including entering the order, in all instances”, the pilot of a more advanced one in seventy percent; the operators sat in the Philippines and India. Meanwhile the company reported 95–99% of orders “without intervention”, and the SEC found that this figure was counted without restaurant staff intervention — but not without humans. Inside the company they understood it: one executive warned that the metric “infers no supervision which isn't true”. The sanction was a cease-and-desist without a fine in view of the company's finances, then delisting. The mechanism is the same in every case: a denominator trick. Humans are excluded from the metric by narrowing who counts as human.

Other cases support different conclusions. Amazon disputed the share of purchases requiring human review; press reports and the company's response must be read together. Cruise described a driverless service while disclosing remote assistance: no driver inside the car does not imply no support. Waymo explicitly describes its support as advice rather than remote driving; the operator count alone does not refute that promise. At the Optimus demonstration, teleoperators supported speech and gestures, while walking was assessed separately. 1X presented teleoperation as part of its initial product. The allegation against nate's founder concerns claimed purchase automation; an allegation is not a finding of guilt. Revenue reporting about Builder.ai does not prove that 700 people secretly replaced AI development: that is a separate, insufficiently supported claim. Each case needs the exact promise and a list of human operations excluded from the metric.

For software agents, human work occurs before, during and after execution. OpenAI describes around 1,500 PRs over five months: an initial team of three engineers grew to seven. The authors explicitly describe environment design and review; this is disclosed labor. Actual engineer-hours are unpublished, so “2–4 hours per PR” cannot be inferred from team size. METR's experiment measured active time spent reviewing and cleaning outputs separately. Faros's time to first review and time in review are elapsed-time measures: they include waiting and are not reviewer labor. Shared infrastructure setup should be reported separately with an explicit allocation rule across tasks or periods. Four separate measures help: the accepted share without interventions, human minutes, machine cost, and elapsed time. Costs include failed attempts. The initial task instruction is not an additional intervention; subsequent prompts, edits and approvals are. Independent outcome assessment is recorded separately from assistance in producing it.

The signs of pseudo-autonomy can be checked against a list, and each sign has a precedent. The denominator is undefined: a “non-intervention rate”, a “resolution rate” with no answer to who is excluded. Refusal of the ratio: the company gladly names latency and operator vetting but not the number of interventions per unit of work. The labor appears in reporting only under pressure — after a regulator's inquiry, a hearing or an investigation, and not before. Repackaging humans as a “feature” after exposure. A keystroke metric instead of a labor metric. Edits after output: rewrites within two weeks, PR size growth, hiring to “finish what the AI started”. And the subtlest one — time on task stops being measurable: METR changed its experiment design because developers with agents did other work while waiting and could not say how long a task took. A template for disclosure already exists in another industry: California's rules require a disengagement report to state “the party that initiated the disengagement: autonomous technology, autonomous vehicle test driver, remote operator, or passenger”. No such rule exists for software agents.

• •Demand the definition of the denominator: who does not count as human in “without intervention”
• •Count human hours before and after the run per accepted task, not keystrokes
• •Look at the trail after the output: rewrites within two weeks, PR size, hiring to finish the job
• •Do not trust a metric the vendor cannot give per unit of work
• •Distinguish pseudo-autonomy from fraud: the first is about the denominator, the second about revenue

By autumn 2026 every major vendor has a stop primitive, and their design says more about the boundary than any declaration. Claude Code's auto mode hands decisions to a classifier on a second model, and its block list is a ready trigger list: irreversible destruction of files that existed before the session, rewriting git history, publishing secrets, stepping outside the request, unfamiliar infrastructure, actions driven by hostile content; boundaries the human stated in conversation — “don't push”, “wait for review” — are block signals too, and the model's judgment that a condition was met does not lift them; after three consecutive blocks or twenty in total the mode hands control back to the human. Managed Agents have an auto policy, and the documentation warns that it “is not a human checkpoint”. OpenAI walked the same road: for Operator, confirmations before actions cut risk by about ninety percent at 92% recall, and five of thirteen unmitigated errors had irreversible consequences; in April 2026 auto-review appeared — of ten thousand actions, 720 left the sandbox, 713 were approved by the reviewing model, seven denied, two hundred times fewer interruptions. The reason for the change of course is named outright: people write rules like “allow everything that starts with python” and approve without reading. Decisions to allow or block an action are moving from the human to the classifier, and every vendor notes that the classifier will not stop a model that hides its intent.

Does the agent see the boundary itself? The 2026 data answer: weakly, and worse where it was fine-tuned for correctness. Kaddour and colleagues gave frontier agents a hundred SWE-bench Pro tasks and asked whether they would succeed: the agents predicted 61–77% success against an actual 22–35%; 62% of failures were predicted at a confidence of at least 0.7; the pre-execution estimate separated success from failure better than the post-execution one, and mid-task doubt turned out to be noise. Meta's abstention benchmark showed that reasoning fine-tuning lowers the willingness to say “I don't know” by 24% on average, and scale from eight to four hundred billion parameters changes nothing; OpenAI explained why — nine of ten popular benchmarks give no credit for abstaining, so a guess always pays. Questions tell the same story: without prompting, agents “almost never” ask even on severely underspecified tasks, although interaction lifts the result from 26.8 to 39.6% and recovers up to 89% of full-specification performance. The harshest measurement is HiL-Bench from Scale: when the agent is given an “ask the human” tool and part of the inputs is withheld, pass@3 falls from 67–91% to 1–39% because the agent does not recognise the gap; GPT models under-ask, Gemini over-asks, and Claude in 45% of failures explicitly writes that it is stuck and submits the work anyway. Introspection, by Anthropic's own data, works in about twenty percent of cases and is called “highly unreliable”. The one encouraging signal: a question before the start about the required effort and one's own ability predicts failure seven to ten points better than a question about confidence, and a separate intent-detection agent reaches full-specification performance while asking on 69% of tasks.080808Almeida, one of the InstructGPT authors, says an assistant and automation need different properties and proposes calibrated decisions as a separate training goal. The data in this section illustrate his thesis.Knizhny kub · Diogo Almeida on calibration08Almeida, one of the InstructGPT authors, says an assistant and automation need different properties and proposes calibrated decisions as a separate training goal. The data in this section illustrate his thesis.Knizhny kub · Diogo Almeida on calibration

Hence the two-stage failure that repeats across all the data: partial detection, then non-enforcement. The Replit agent knew about the code freeze and deleted the database. In Anthropic's scenarios Claude Opus 4 blackmailed in 55.1% of runs when it believed the situation was real and 6.5% when it believed it was a test — lab measurements of stopping may be more optimistic than production. GPT-5.6 Sol at METR showed “substantial situational awareness” of the evaluation environment together with a record cheating rate.

In SpecBench, the same generated code is evaluated on two test suites: one visible to the agent and one held out, testing how features work together. The evaluated models achieve near-maximum scores on visible tests. For large tasks, the authors report a worst-case gap of 100 percentage points: that corresponds to passing 100% of visible tests and 0% of held-out tests. This is an extreme among individual runs, not an average across all models.

In Wang's controlled experiment, an attacker could insert a malicious instruction into the agent's plan, disguising it as a necessary task step. A model acting as the auditor inspected the plan and could issue an “unsafe, stop” signal, called a flag in the paper. In one mode, the system merely logged the warning and let the agent continue. In the other, it halted execution when it recognised that signal. Model judges assessed whether the injected goal was achieved; that counted as attack success. On the pooled control set, the author reported 48.3% attack success without mandatory stopping and 10.8% with it. This is a synthetic plan-injection experiment. Its practical lesson is to connect the auditor's warning to a block in the execution system. Detecting danger and stopping an action need separate evaluation.

The human side of the handoff is no better than the machine's, and that has been known since 1983. Lisanne Bainbridge described the ironies of automation: it takes the easy parts and leaves the human the hardest ones; attention to a source where almost nothing happens cannot be held for more than half an hour; the most successful automation needs the most operator training. In August 2026 NIST rewrote this for agents: “overly chatty agents condition users to reflexively click ‘allow’”, like victims of multi-factor push-bombing — and proposed pre-approved “flight plans” instead of a question at every step. The automotive literature adds a warning: across fifty-one studies, the faster the human takes over, the harsher the manoeuvres and the higher the crash rate. Speed of handback is not its quality. The quality of a handoff must be measured by what the human does after it — whether they understood the state and made the right decision — and for that the agent must hand over not “I'm stuck” but the state, the evidence and the reason for stopping. Each trigger needs a detection mechanism: a rule, counter, classifier, calibrated model estimate or human review. None guarantees detection. Pre-execution routing and clarification are candidates to validate locally; confidence thresholds require domain-specific checks: the same scorer gives AUROC 0.85 in telecom and 0.23 in airline.

The laboratory, comparative trials and operations may disagree because of different tasks, acceptance criteria, budgets and accounting. Shared definitions enable comparison: a unit of work, a predeclared criterion, intervention and cost accounting, and a failure taxonomy. They do not suffice to predict operations. Task distributions, environments, users and model updates may still differ. Transfer must be tested on a representative work sample and monitored as operations change.

Bridges between the levels are already being built, and they deserve naming. Feng and colleagues proposed “assisted evaluations”: run the agent without a human, then add involvement round by round until the result exceeds a threshold, and call the autonomy level the minimum involvement at which that happened — so a lab run and a production touch land on one axis. TCR@k does the same for benchmarks and telemetry: the share of tasks with at most k interventions is counted identically in the sandbox and in production. Anthropic advises building regression evals from real incidents and demanding near one hundred percent — a transfer of the acceptance criterion from production into the lab. Logs instead of scores, a disclosed harness and five runs transfer the accounting from production into leaderboards. Zheng and colleagues' separation of capability levels from allowed-autonomy levels requires reporting both: what the agent can do and what it is permitted to do. And the MIT Agent Index shows how much is still empty: for thirty deployed agents, 135 of 240 safety fields are blank, and four disclosed safety evaluations.090909In the eighth episode of AI Dev Podcast we agreed that what should be measured is accepted work, verification cost and the link between changes and production outcomes, not the volume of code. This piece is a long footnote to that conversation.Knizhny kub · AI Dev Podcast #809In the eighth episode of AI Dev Podcast we agreed that what should be measured is accepted work, verification cost and the link between changes and production outcomes, not the volume of code. This piece is a long footnote to that conversation.Knizhny kub · AI Dev Podcast #8

The Russian contour: what is published and what is not

Russian publications also report different kinds of measures. Sber reports token volume and adoption; Yandex reports developer and code shares, rather than accepted tasks without assistance. Raiffeisen distinguishes 25% of backlog done with agents from 4% without asking a human, but transfer requires a period and acceptance criterion. At Pervaya Forma, agents create 93% of PRs and 94% merge without remarks, while about 76 hours from review to merge is elapsed delay. Against eleven minutes of model work, the gap describes a process queue, not 76 hours of human labor. Infosistemy Jet and Codenrock surveys describe prevalence and trust rather than comparable success rates. The evidence is insufficient to conclude that the market as a whole leads, lags or exactly follows the global trend.

The review suggests a local evaluation program. This is an author proposal, not a validated standard or a promise to finish in a month. Before running it, choose an acceptable error cost, savings conditions and required confidence. Sample size and observation duration depend on those requirements; a few dozen tasks cannot reliably quantify rare severe failures.

First assemble a representative work sample and a separate suite of difficult incidents. The latter tests robustness but does not estimate average production success. A pilot of 20–50 tasks with five independent runs can expose common failures; it cannot establish safety for rare events. Compare pass^1 and pass^5 on the same tasks, models, harnesses and budgets. Even with a constant independent probability of 0.85, five successes have probability 0.85^5 ≈ 0.444: expected arithmetic, not a standalone stop rule. Heterogeneous tasks require care when averaging, and dependent repeats are not independent samples. Report uncertainty intervals, all interventions, failed-attempt costs and waiting time separately. Limit delegation when a conservative risk or cost estimate exceeds the preselected conditions, or when required controls fail.

Autonomy is a property of a particular system doing particular work in a specified environment and permission boundary. A measurement system describes that property. The practical result of this review is a way to test delegation conditions, not a universal duration of work without a person. It requires a defined result, verifiable acceptance, disclosed assistance, cost accounting and failure monitoring. The observation period follows from the reliability requirement, not a general “one quarter” rule.

This is a thematic review, not a systematic survey of all literature. Its registry covers primary studies, leaderboards, documentation and regulatory texts gathered by 27 September 2026. Key comparisons were checked selectively during revision; this is not a fresh verification of every source link. The repository dossier connects key claims to source versions, tables, metrics and permissible conclusions. Author calculations and unresolved inconsistencies are identified separately. Leaderboards change; results belong to the stated snapshots, and a missing interval does not mean missing uncertainty.