What Can We Trust an AI Agent With? Boundaries of Autonomy

Alexander proposes a working ladder: an action, a task, a block of work, a role, and an end-to-end outcome. Calling a tool, handling a support request, delivering a feature, and replacing a role involve different commitments. This is a way to frame the conversation, not a universal classification. Aleksey adds another distinction: a specific task, a direction for improvement, and an area of responsibility in which tasks must first be identified. Calling an agent a business analyst does not establish how its work will finish or be accepted.

Autonomy also depends on the environment and permissions. A new company can design around agents from the outset; an established one must account for existing interfaces, human handoffs, and constraints. The robotics analogy makes this concrete: a prepared indoor environment differs from changing outdoor conditions. The agreement therefore needs to specify the work, permitted actions, and operating conditions. Maximum independence is not an objective in itself. An intermediate level may meet the business need.

Evgeny describes content releases at Flo Health: uploading material, configuring an experiment, releasing a change, checking logs, and announcing the launch. Automating individual steps gradually moves the person toward managing the system and identifying where intervention remains necessary. Automatic detection and repair of problems is discussed as a direction, rather than an achieved endpoint. Verification effort matters too: if confirming the result consumes much of the time saved, agent independence has limited value.

Alexander gives a marketing example where legal and brand approvals, rather than writing, constrained delivery. Faster generation merely lengthened the queue. Evgeny describes a related move from separate expert-written evaluation prompts toward explicit policies: independent instructions duplicate logic and eventually contradict one another. Requirements need agreement and should move earlier in the process, allowing the agent to consider them before producing the final material. The relevant improvement covers the entire path to an accepted result.

Aleksey experiments with fewer instructions after a new model arrives. He simplifies the task description, observes failures, and restores only the rules that remain necessary. This tests how much former preparation the model can now handle. Evgeny likewise prefers starting with a simple process and adding constraints after specific failures. The hosts describe complementary activities: strengthen the system based on mistakes, then remove obsolete complexity as model capabilities change.

These experiments need a measurement baseline: representative tasks, success criteria, and previous results for comparison. Otherwise, shorter instructions can look like progress while quality deteriorates. The hosts distinguish finding one successful attempt from obtaining consistently successful results. Multiple attempts help when a reliable check can identify the winner. Repeatability requires more than a convincing demonstration. Human questions and manual corrections must be counted separately because they reveal how independently the work actually proceeds.

Evgeny describes porting his game from iOS to Android. Instead of prescribing an implementation, he supplied the existing project and asked the agent to compare screens and mechanics through both platforms’ emulators. He reports completion in roughly five hours without intervention. The existing game provided a testable reference. Alexander emphasizes the limits: the experiment did not require immediate store publication or responsibility toward users. Success in a personal project does not establish that the same process is ready for a production system.

Alexander’s podcast-cover example shows another boundary. He and the agent established visual references and a repeatable process; his remaining work largely involved locating suitable guest photographs. The agent requested material it could not reliably obtain itself. Cost and maintenance constraints also matter: the author wants to discuss useful capabilities while reducing technical routine. A company may optimize for different requirements. Formal human approval cannot substitute for actual control over consequences.

Discussing METR’s approach, Alexander separates agent runtime from the time a comparable task would take a human. A task horizon relates human task duration to the agent’s probability of success. A long-running session therefore establishes neither the difficulty of the problem nor its economic value. An evaluation at a particular success threshold also does not establish sufficient reliability for a specific workflow. The hosts return to the company’s own tasks and checks as the basis for delegation decisions.

Long action sequences create further opportunities for failure, but the hosts describe simple multiplication of independent error probabilities as an inadequate model. A system can preserve intermediate states, return to them, and try another route. Evaluation must therefore cover the model together with its tools and recovery mechanisms. Public benchmarks provide useful reference points, but they cannot account for your project’s conditions, data, or acceptable cost of failure.

Aleksey asks how much an agent needs to know about the purpose behind a task. For the game migration, Evgeny had a sufficient reference already; elsewhere, the reasons behind constraints matter. An architecture decision record preserves the options considered and why some were rejected. People learn this context through conversations. A new agent cannot recover it if it was never recorded. Reviewing a plan both aligns intentions and leaves knowledge for subsequent work.

Excessive detail can also prevent a better solution from emerging. Aleksey suggests defining the desired outcome and meaningful boundaries while retaining room for exploration. Evgeny connects this to Regenerative Software: one proposed test of sufficient system knowledge is regenerating part of the implementation from its behavior and constraints. The discussion treats this as a direction for development, not an established universal practice. A process that still needs constant manual corrections remains short of autonomy.

Alexander brings several measurements together: accepted tasks, human interventions, verification effort, recovery time, the scope of possible damage, and cost per accepted result. Failed attempts belong in those costs. Aleksey asks whether human effort has simply moved into preparation and repair. The hosts include people maintaining evaluations, updating policies, and taking control when something fails. A recurring exception is effectively part of the workflow and must be reflected in its claimed independence.

Evgeny suggests using session logs to find repeated mistakes and improve the process. He describes a Flo Health experiment that analyzes multiple sessions to identify common difficulties and propose changes. The closing discussion turns to management. Engineers increasingly need delegation and attention-management skills; Evgeny still sees a role for managers in coordinating teams and developing people. Aleksey expects flatter structures. The hosts do not reach a single forecast: organizational scale and design remain important conditions.