AI Development as a Co-Evolving Stack: Hardware, Models, Harnesses, Tools, and Traces

The formula Agent = Model + Harness is useful as a first approximation and dangerous as a stopping point. It hides inference economics, the actual action environment, identity, outcome verification, and the feedback loop. Two teams can buy the same checkpoint and build very different products: one supplies a familiar patch primitive, a bounded shell, and compact tool results; the other exposes a hundred novel JSON schemas, noisy context, and approval prompts for every safe step.010101“Stack” sounds too static. It is more useful to think in terms of several control loops that share signals while updating at different speeds.

The system has at least six layers, each answering a different question. A model score describes capability in a particular benchmark. The harness defines available context and recovery. The execution environment turns intent into a verifiable change. Evals show whether quality survived a release. Traces provide diagnostic material—but not permission to train.

The inner loop runs in days and weeks: teams change tool descriptions, context packaging, budgets, routing, and release gates. The outer loop runs in months and years: post-training, serving architecture, accelerators, and network fabric. A local failure does not always require a new model. When the same failure class repeats across thousands of episodes, however, the signal can move upstream into training, runtime, or the next hardware generation.

An accelerator is more than a FLOPS count. In Hopper, Transformer Engine links FP8/FP16 to transformer workloads while NVLink changes the price of inter-GPU communication. Blackwell extends the idea to a rack-scale system, and the Vera Rubin announcement explicitly uses the term extreme codesign: CPU, GPU, interconnect, storage, and networking are presented as one system. Acceleration and TCO figures on those pages are vendor claims; the direction of design is directly observable.

The clearest shift is the divergence of training and inference. Google’s TPU 8t and TPU 8i are two systems: one optimized for large-scale pretraining, the other for serving, sampling, and long reasoning workloads. Memory, on-chip SRAM, networking, and collectives change with MoE and agentic workloads. For a CTO, this means the “per token” price hides provider decisions about batch, cache, latency, and available capacity.

A full vertical stack is not the only route. DeepSeek-V3 demonstrates reverse co-design: the team adapted FP8, MoE, expert balancing, and its training framework to available H800 hardware. The GPU-hours remain self-reported, but the architectural response to the constraint is documented. Anthropic and Annapurna Labs show another configuration: a dense partner loop without owning the hyperscaler.

Function calling tests whether a model can form a call. A real agent must also select the tool, interpret errors, continue after a large result, verify effects, and stay within its authority. These habits emerge from the distribution of action spaces seen during post-training.

Cursor describes spending weeks adapting its harness to each new model: a familiar patch-based edit for OpenAI models and string replacement for Anthropic models. In a separate Codex account, the team changed tool names, the preamble, lint feedback, and reasoning-trace handling because the model had shell-first behavior. These are company observations rather than an open A/B dataset, but they align with the nature of post-training.

A harness is necessary not because the model is “not smart enough,” but because autonomous work requires a protocol. The Codex App Server includes lifecycle and persistent threads, config/auth, sandboxing, MCP, and skills under one policy model. In Anthropic’s long-running agent experiments, a high-level prompt plus compaction was insufficient: the agent needed an initializer, a feature list, progress artifacts, git history, and small iterations.020202A practical definition I like: a harness is a temporary theory of the current model’s weaknesses. The danger lies in treating that theory as permanent.

Responsibility moves as reliability improves. Cursor once rechecked every task, forced commit/push, and fetched CI logs through hardcoded harness logic. Those actions later became tools controlled by the model. At the same time, weaker computer-use capability still required dedicated routing, a subagent, and screen recording. The harness does not “end”: it stops constraining a mature capability and starts protecting a new one.

MCP solves an important but narrower problem. The tools specification standardizes discovery, schemas, and call semantics. It does not promise that a model saw a similar schema in post-training, will choose correctly among dozens of tools, receive a compact response, or act under the right identity.

A large catalog can be worse than a small set of familiar primitives. In the code execution with MCP case, Anthropic shows two costs of direct connection: schemas occupy context and intermediate results repeatedly pass through the model. A meeting-transcript example added roughly 50,000 tokens. Progressive discovery and code execution keep large data inside a sandbox and return only the necessary result. That does not make shell safe by default: composability adds package, network, and secrets-policy risk.

An internal tool gateway is therefore not a plumbing project. It is a behavioral product for the platform team. Its moat is not the number of connected servers, but the quality of contracts and the evidence about how agents fail in selection, arguments, refusal, and recovery.

“Trace” is often used as shorthand for magical self-improvement. It is better to separate four modes. Infrastructure telemetry improves serving and timeouts. Product telemetry surfaces tool-selection errors, permission stops, and compaction failures. Outcome evals connect episodes to tests, merges, reverts, or user corrections. A training trajectory is a deliberately selected and permitted rollout with a reward or verifier.

The distinction between trajectory and outcome is essential. In Anthropic’s eval methodology, a transcript is the sequence of steps while the outcome is the state of the environment afterward. τ-bench adds reliability across repeated runs: a single success is not evidence of stability. GitHub separately evaluates tool selection and arguments for its MCP Server, while Databricks connects production-like scenarios, traces, and regression gates. All of these can improve a system without changing model weights.

Even when a provider trains a model in a realistic environment, it is wrong to attribute training to all customer sessions. Cursor Privacy Mode excludes customer data from training; Anthropic separates commercial sessions from opt-in programs; and OpenAI does not train on business/API inputs and outputs by default. Observability, eval use, and model training require separate policies, retention rules, and legal bases.030303The most expensive lock-in often lives outside the model API: in a closed format for episodes and outcomes that cannot be replayed with another provider.

production failure → trace + end state → reproducible eval → harness/tool/model change → repeated gate → controlled rollout

A common telemetry schema, such as the evolving OpenTelemetry GenAI conventions, lowers export cost. Field names still do not decide retention, redaction, or training rights. Those remain the system owner’s responsibility.

The original hypothesis was intentionally provocative: within 180 ± 60 days, half of a meaningful harness becomes unnecessary, replaced, or removed. Line churn cannot test it: a large UI diff may leave agent behavior unchanged. The research therefore fixed ten mechanisms—context, compaction, planning, edit, policy, discovery, sandbox, persistence, orchestration, and telemetry/evals—and inspected the public histories of Codex, Gemini CLI, and OpenCode.

At least seven mechanisms changed substantially in every project over six months. Under a strict criterion—the old behavioral path is actually retired rather than coexisting behind a feature flag—clear replacement appears in roughly three to four out of ten. The research did not find two independent projects with proven ≥ 5/10.

The uncertainty is epistemic rather than statistical. Git history does not always reveal rollout percentage, deployment status, or the role of feature flags. Young projects also inflate visible change. The strategic conclusion is still robust: a proprietary generic harness is not a one-off build but an indefinite compatibility program.

Integration accelerates feedback. A team that can see model behavior, product traces, and outcomes can connect a regression to the right layer faster. The fixed cost of frontier training, multi-OS sandboxing, eval farms, and enterprise governance strengthens scale effects. It still does not follow that one company must own and win every layer.

Three routes to a short cycle

• —Google connects TPU, networking, compiler, model, Cloud, and distribution—a full upstream vertical.
• —Anthropic owns the model and harness while building hardware co-design as a partnership with AWS.
• —Cursor started with developer workflow and distribution, accumulated traces and evals, then added post-training and a model layer.

Counterforces are real. Open weights from DeepSeek, Qwen, and GLM lower switching barriers. Qwen Code and OpenCode move session UX, permissions, and some tools outside a single lab’s control. MCP and compatible APIs reduce integration cost. RouteLLM demonstrates the principle of routing between stronger and cheaper models. And a corporate environment may be so specific that a generic provider sees neither authority nor the final outcome.

Several ecosystems with specialized fast paths are therefore more likely than one universal runtime. The moat forms where an organization controls several adjacent layers and has an evidence channel between them. Owning everything is only one configuration.

It is more useful to draw the boundary by rate of change and organizational uniqueness than by provider logo. Frontier capability changes quickly and is rarely company-specific. Identity, policy, domain action contracts, and accepted outcomes age more slowly and determine what the system is even allowed to count as success.

What not to build by default

• —another general coding-agent loop merely to swap an API endpoint;
• —custom compaction or orchestration without a regression suite built from real episodes;
• —a catalog of hundreds of MCP tools without owners, response limits, or selection telemetry;
• —a “self-learning” system that automatically treats operational logs as permitted training data;
• —a gateway that normalizes every provider to the lowest common denominator and hides their strengths;
• —a fork of an open-source harness without a team responsible for upstream merges, security, and model compatibility.

A proprietary harness is justified when several conditions coincide: a unique action environment, sovereignty or extreme latency constraints, enough volume, mature evals, and willingness to maintain model-specific fast paths. The strongest condition is that agent capability is part of the external product and the strategic moat. A talented platform team alone is not enough.

An enterprise can implement co-design without owning an accelerator or a foundation model. It needs to close the part of the cycle it actually controls: its tasks, authority, tools, end states, and rollout decision.

A minimum one-quarter program

1. 01Select 20–30 real episodes. Freeze a clean start state, task contract, permitted tools, and a verifiable outcome.
2. 02Separate trace modes. Define operational telemetry, human review, eval use, and permitted training trajectories independently.
3. 03Inventory action contracts. For each critical tool, record identity, scope, idempotency, response limit, provenance, and owner.
4. 04Build provider-neutral replay. Store the episode, environment version, model/harness/tool versions, and end-state evidence in a portable format.
5. 05Introduce repeated release gates. Compare quality, variance, cost, safety, and human acceptance rather than one successful run.
6. 06Test the layer, not the brand. Change a tool description, context, policy, routing, or model only through a controlled experiment.

In this loop, the provider supplies rapidly changing capability while the enterprise retains release authority. A new model may be cheaper and stronger on public benchmarks; production rollout still passes internal episodes, policy checks, and comparison of accepted outcomes. That is the practical defense against both technical and evidentiary lock-in.