AI Development as a Co-Evolving Stack: Hardware, Models, Harnesses, Tools, and Traces

The formula Agent = Model + Harness is useful as a first approximation and dangerous as a stopping point. It hides inference economics, the actual action environment, identity, outcome verification, and the feedback loop. Two teams can buy the same checkpoint and build very different products: one supplies a familiar patch primitive, a bounded shell, and compact tool results; the other exposes a hundred novel JSON schemas, noisy context, and approval prompts for every safe step.010101“Stack” sounds too static. It is more useful to think in terms of several control loops that share signals while updating at different speeds. One live example of such a loop is an internal agent that runs model experiments on its own: memory of past runs, budgets, runbooks for common failures, and control points. I walked through its design in my channel.Knizhny kub · an agent that closes the ML experiment loop (in Russian)Knizhny kub · how that agent is built (in Russian)01“Stack” sounds too static. It is more useful to think in terms of several control loops that share signals while updating at different speeds. One live example of such a loop is an internal agent that runs model experiments on its own: memory of past runs, budgets, runbooks for common failures, and control points. I walked through its design in my channel.Knizhny kub · an agent that closes the ML experiment loop (in Russian)Knizhny kub · how that agent is built (in Russian)

The system has at least six layers, each answering a different question. A model score describes capability in a particular benchmark. The harness defines available context and recovery. The execution environment turns intent into a verifiable change. Evals show whether quality survived a release. Traces provide diagnostic material—but not permission to train.

The inner loop runs in days and weeks: teams change tool descriptions, context packaging, budgets, routing, and release gates. The outer loop runs in months and years: post-training, serving architecture, accelerators, and network fabric. A local failure does not always require a new model. When the same failure class repeats across thousands of episodes, however, the signal can move upstream into training, runtime, or the next hardware generation.

An accelerator is more than a FLOPS count. In Hopper, Transformer Engine links FP8/FP16 to transformer workloads while NVLink changes the price of inter-GPU communication. Blackwell extends the idea to a rack-scale system, and the Vera Rubin announcement explicitly uses the term extreme codesign: CPU, GPU, interconnect, storage, and networking are presented as one system. Acceleration and TCO figures on those pages are vendor claims; the direction of design is directly observable.020202The idea is not new: Jeff Dean names hardware specialization and sparse models as conditions for the next step, and the NVIDIA story shows that AI infrastructure is a system of systems, where memory, fabric, and the software layer matter as much as FLOPS. I retold both in my channel.Knizhny kub · Jeff Dean on how we got here (in Russian)Knizhny kub · NVIDIA as a system of systems (in Russian)02The idea is not new: Jeff Dean names hardware specialization and sparse models as conditions for the next step, and the NVIDIA story shows that AI infrastructure is a system of systems, where memory, fabric, and the software layer matter as much as FLOPS. I retold both in my channel.Knizhny kub · Jeff Dean on how we got here (in Russian)Knizhny kub · NVIDIA as a system of systems (in Russian)

The clearest shift is the divergence of training and inference. Google’s TPU 8t and TPU 8i are two systems: one optimized for large-scale pretraining, the other for serving, sampling, and long reasoning workloads. Memory, on-chip SRAM, networking, and collectives change with MoE and agentic workloads. For a CTO, this means the “per token” price hides provider decisions about batch, cache, latency, and available capacity.

A full vertical stack is not the only route. DeepSeek-V3 demonstrates reverse co-design: the team adapted FP8, MoE, expert balancing, and its training framework to available H800 hardware. The GPU-hours remain self-reported, but the architectural response to the constraint is documented. Anthropic and Annapurna Labs show another configuration: a dense partner loop without owning the hyperscaler.

Function calling tests whether a model can form a call. A real agent must also select the tool, interpret errors, continue after a large result, verify effects, and stay within its authority. These habits emerge from the distribution of action spaces seen during post-training.

Cursor describes spending weeks adapting its harness to each new model: a familiar patch-based edit for OpenAI models and string replacement for Anthropic models. In a separate Codex write-up, the team changed tool names, the preamble, lint feedback, and reasoning-trace handling because the model had shell-first behavior. These are company observations rather than an open A/B dataset, but they align with the nature of post-training.

A harness is necessary not because the model is “not smart enough,” but because autonomous work requires a protocol. The Codex App Server includes lifecycle and persistent threads, config/auth, sandboxing, MCP, and skills under one policy model. In Anthropic’s long-running agent experiments, a high-level prompt plus compaction was insufficient: the agent needed an initializer, a feature list, progress artifacts, git history, and small iterations.030303A practical definition I like: a harness is a temporary theory of the current model’s weaknesses. The danger lies in treating that theory as permanent. Vendors present it as one bundle anyway: the Gemini 3.5 announcement ships an updated harness together with the model and bets on long tasks and subagents. I covered that announcement in my channel.Knizhny kub · Gemini 3.5 and the bet on agents (in Russian)03A practical definition I like: a harness is a temporary theory of the current model’s weaknesses. The danger lies in treating that theory as permanent. Vendors present it as one bundle anyway: the Gemini 3.5 announcement ships an updated harness together with the model and bets on long tasks and subagents. I covered that announcement in my channel.Knizhny kub · Gemini 3.5 and the bet on agents (in Russian)

Responsibility moves as reliability improves. Cursor once rechecked every task, forced commit/push, and fetched CI logs through hardcoded harness logic. Those actions later became tools controlled by the model. At the same time, weaker computer-use capability still required dedicated routing, a subagent, and screen recording. The harness does not “end”: it stops constraining a mature capability and starts protecting a new one.

MCP solves an important but narrower problem. The tools specification standardizes discovery, schemas, and call semantics. It does not promise that a model saw a similar schema in post-training, will choose correctly among dozens of tools, receive a compact response, or act under the right identity.040404Connecting a tool is the easy part. Dex Horthy shows how a tool catalog fills the context with schemas and identifiers and drags the agent into a degradation zone, while recent agent transport protocol drafts try to raise identity and authority to the protocol itself. I covered both in my channel.Knizhny kub · No Vibes Allowed on context engineering (in Russian)Knizhny kub · transport protocols for agents (in Russian)04Connecting a tool is the easy part. Dex Horthy shows how a tool catalog fills the context with schemas and identifiers and drags the agent into a degradation zone, while recent agent transport protocol drafts try to raise identity and authority to the protocol itself. I covered both in my channel.Knizhny kub · No Vibes Allowed on context engineering (in Russian)Knizhny kub · transport protocols for agents (in Russian)

A large catalog can be worse than a small set of familiar primitives. In the code execution with MCP case, Anthropic shows two costs of direct connection: schemas occupy context and intermediate results repeatedly pass through the model. A meeting-transcript example added roughly 50,000 tokens. Progressive discovery and code execution keep large data inside a sandbox and return only the necessary result. That does not make shell safe by default: composability adds package, network, and secrets-policy risk.

An internal tool gateway is therefore not a plumbing project. It is a behavioral product for the platform team. Its moat is not the number of connected servers, but the quality of contracts and the evidence about how agents fail in selection, arguments, refusal, and recovery.

“Trace” is often used as shorthand for magical self-improvement. It is better to separate four modes. Infrastructure telemetry improves serving and timeouts. Product telemetry surfaces tool-selection errors, permission stops, and compaction failures. Outcome evals connect episodes to tests, merges, reverts, or user corrections. A training trajectory is a deliberately selected and permitted rollout with a reward or verifier.

The distinction between trajectory and outcome is essential. In Anthropic’s eval methodology, a transcript is the sequence of steps while the outcome is the state of the environment afterward. τ-bench adds reliability across repeated runs: a single success is not evidence of stability. GitHub separately evaluates tool selection and arguments for its MCP Server, while Databricks connects production-like scenarios, traces, and regression gates. All of these can improve a system without changing model weights.

Even when a provider trains a model in a realistic environment, it is wrong to attribute training to all customer sessions. Cursor Privacy Mode excludes customer data from training; Anthropic separates commercial sessions from opt-in programs (policies as reviewed in July 2026); and OpenAI does not train on business/API inputs and outputs by default. Observability, eval use, and model training require separate policies, retention rules, and legal bases.050505The most expensive lock-in often lives outside the model API: in a closed format for episodes and outcomes that cannot be replayed with another provider. And defaults shift under you: since April 2026 GitHub may use Copilot interaction data from personal Free, Pro, and Pro+ plans for model training unless it is turned off by hand; Business and Enterprise accounts stay excluded. I covered that shift in my channel.Knizhny kub · GitHub trains on interaction data (in Russian)05The most expensive lock-in often lives outside the model API: in a closed format for episodes and outcomes that cannot be replayed with another provider. And defaults shift under you: since April 2026 GitHub may use Copilot interaction data from personal Free, Pro, and Pro+ plans for model training unless it is turned off by hand; Business and Enterprise accounts stay excluded. I covered that shift in my channel.Knizhny kub · GitHub trains on interaction data (in Russian)

production failure → trace + end state → reproducible eval → harness/tool/model change → repeated gate → controlled rollout

A common telemetry schema, such as the evolving OpenTelemetry GenAI conventions, lowers export cost. Field names still do not decide retention, redaction, or training rights. Those remain the system owner’s responsibility.

The original hypothesis was intentionally provocative: within 180 ± 60 days, half of a meaningful harness becomes unnecessary, replaced, or removed. Line churn cannot test it: a large UI diff may leave agent behavior unchanged. The research therefore fixed ten mechanisms—context, compaction, planning, edit, policy, discovery, sandbox, persistence, orchestration, and telemetry/evals—and inspected the public histories of Codex, Gemini CLI, and OpenCode.

At least seven mechanisms changed substantially in every project over six months. Under a strict criterion—the old behavioral path is actually retired rather than coexisting behind a feature flag—clear replacement appears in roughly three to four out of ten. The research did not find two independent projects with proven ≥ 5/10.

The uncertainty is epistemic rather than statistical. Git history does not always reveal rollout percentage, deployment status, or the role of feature flags. Young projects also inflate visible change. The strategic conclusion is still robust: a proprietary generic harness is not a one-off build but an indefinite compatibility program.

Integration accelerates feedback. A team that can see model behavior, product traces, and outcomes can connect a regression to the right layer faster. The fixed cost of frontier training, multi-OS sandboxing, eval farms, and enterprise governance strengthens scale effects. That increases concentration inside individual loops, but it still does not imply that one company must own and win every layer.

Four ways to close the short loop

• —Google connects TPU, networking, compiler, model, Cloud, and distribution—a full upstream vertical.
• —Anthropic owns the model and harness while building hardware co-design as a partnership with AWS.
• —Cursor started with developer workflow and distribution, accumulated traces and evals, then added post-training and a model layer.
• —OpenAI is building a hybrid partner-led vertical: Stargate connects the lab to clouds, data centers, energy, and capital, while its work with Broadcom adds OpenAI-designed accelerators and networking systems. It does not own every physical layer, but it shapes the loop’s architecture and carries model and product knowledge into hardware.

The China path: a national loop built on domestic hardware

China’s shift toward domestic hardware moves the short loop to the scale of a national technology ecosystem. The authors of Pangu Ultra report pretraining a dense 135-billion-parameter model on 13.2 trillion tokens using 8,192 Ascend NPUs. In Huawei’s roadmap, Ascend 950PR targets prefill and recommendation while Ascend 950DT targets decode and training. Interconnect, low-precision formats, CANN, tooling, and openPangu evolve around them. This is a verifiable example of training a large model on a Chinese stack, not proof that the entire Chinese market has moved away from NVIDIA.

The evidence boundary around DeepSeek V4 is narrower. In its official release, DeepSeek lists V4-Pro at 1.6 trillion total and 49 billion active parameters, and V4-Flash at 284 billion total and 13 billion active parameters, but does not disclose the hardware used for base pretraining. The SLAI T-Rex study does document full-parameter post-training of the V4 family on an Ascend NPU SuperPOD: the authors report 34.22% MFU and a 2.93× improvement over the open baseline recipe, and describe continued pretraining and SFT for V4-Flash. That is verified trillion-parameter-scale training on Ascend, but it is not evidence that the base V4 model was pretrained from scratch on those accelerators.

The position of Chinese big labs in 2026 is neither an absolute ban on NVIDIA nor a simple refusal to buy it. BIS moved H200 applications to case-by-case review, but NVIDIA itself reported authorization for only small volumes to specific customers, no H200 program revenue, and effective foreclosure from China’s data-center compute market. At the same time, some labs are making a strategic choice: Reuters reported that DeepSeek gave domestic chipmakers several weeks of early V4 access for optimization while withholding it from NVIDIA and AMD. After the release, Reuters also reported new Ascend 950 inquiries from ByteDance, Tencent, and Alibaba amid constrained supply until shipments scale in the second half of 2026. The companies did not publicly confirm those procurement talks, so this is an agency-sourced market snapshot, not an official industry-wide rejection of NVIDIA.

Counterforces: open models and routing

Vertical integration is not the only force. Open weights from DeepSeek, Qwen, and GLM lower switching barriers. Qwen Code and OpenCode move session UX, permissions, and some tools outside a single lab’s control. MCP and compatible APIs reduce integration cost. RouteLLM formalizes the choice between a strong expensive model and a cheaper one, while OpenRouter Auto Router turns the same principle into a product: it classifies the task, selects from an allowed model pool, applies a cost-quality trade-off, and preserves fallback routes. It does not reproduce RouteLLM’s specific learning algorithm; it implements the same architectural idea across models from different providers. A corporate environment may still be so specific that a generic provider sees neither authority nor the final outcome.060606Marc Andreessen looks at the same picture from the other side: a handful of large models on top, a cascade of cheap ones below, open weights out of China, and products that juggle dozens of models at once. I covered his outlook in my channel.Knizhny kub · Andreessen on China and the price of AI (in Russian)06Marc Andreessen looks at the same picture from the other side: a handful of large models on top, a cascade of cheap ones below, open weights out of China, and products that juggle dozens of models at once. I covered his outlook in my channel.Knizhny kub · Andreessen on China and the price of AI (in Russian)

Several ecosystems with specialized fast paths are therefore more likely than one universal runtime. The moat forms where an organization controls several adjacent layers and has an evidence channel between them. Owning everything is only one configuration.

It is more useful to draw the boundary by rate of change and organizational uniqueness than by provider logo. Frontier capability changes quickly and is rarely company-specific. Identity, policy, domain action contracts, and accepted outcomes age more slowly and determine what the system is even allowed to count as success.

What not to build by default

• —another general coding-agent loop merely to swap an API endpoint;
• —custom compaction or orchestration without a regression suite built from real episodes;
• —a catalog of hundreds of MCP tools without owners, response limits, or selection telemetry;
• —a “self-learning” system that automatically treats operational logs as permitted training data;
• —a gateway that normalizes every provider to the lowest common denominator and hides their strengths;
• —a fork of an open-source harness without a team responsible for upstream merges, security, and model compatibility.

A proprietary harness is justified when several conditions coincide: a unique action environment, sovereignty or extreme latency constraints, enough volume, mature evals, and willingness to maintain model-specific fast paths. The strongest condition is that agent capability is part of the external product and the strategic moat. A talented platform team alone is not enough.070707I reached a similar conclusion about the development platform: once agents enter the main flow, code review, security checks, policies, and telemetry stop being external scaffolding and become part of the compute path. I wrote about it separately.Knizhny kub · from AI-native development to an AI-native platform (in Russian)07I reached a similar conclusion about the development platform: once agents enter the main flow, code review, security checks, policies, and telemetry stop being external scaffolding and become part of the compute path. I wrote about it separately.Knizhny kub · from AI-native development to an AI-native platform (in Russian)

An enterprise can implement co-design without owning an accelerator or a foundation model. It needs to close the part of the cycle it actually controls: its tasks, authority, tools, end states, and rollout decision.

A minimum one-quarter program

1. 01Select 20–30 real episodes. Freeze a clean start state, task contract, permitted tools, and a verifiable outcome.
2. 02Separate trace modes. Define operational telemetry, human review, eval use, and permitted training trajectories independently.
3. 03Inventory action contracts. For each critical tool, record identity, scope, idempotency, response limit, provenance, and owner.
4. 04Build provider-neutral replay. Store the episode, environment version, model/harness/tool versions, and end-state evidence in a portable format.
5. 05Introduce repeated release gates. Compare quality, variance, cost, safety, and human acceptance rather than one successful run.
6. 06Test the layer, not the brand. Change a tool description, context, policy, routing, or model only through a controlled experiment.

In this loop, the provider supplies rapidly changing capability while the enterprise retains release authority. A new model may be cheaper and stronger on public benchmarks; production rollout still passes internal episodes, policy checks, and comparison of accepted outcomes. That is the practical defense against both technical and evidentiary lock-in.080808My “State of AI4SDLC” talk made the same argument: adopting AI changes the production system of development rather than adding a tool, and the chain intent → context → plan → verification says more about outcomes than usage metrics. The recording is in my channel.Knizhny kub · the State of AI4SDLC talk (in Russian)08My “State of AI4SDLC” talk made the same argument: adopting AI changes the production system of development rather than adding a tool, and the chain intent → context → plan → verification says more about outcomes than usage metrics. The recording is in my channel.Knizhny kub · the State of AI4SDLC talk (in Russian)