How Vercel Built d0: An Agent's Evolution and Questions About Its Results

In “How We Solved Agent Building,” Andrew Qu, Chief of Software at Vercel, describes how the team built its internal analytics agent, d0. Each architectural decision ran into the next problem. According to Qu, the result was a useful agent and, later, the eve framework. How far that journey supports the word “solved” in the talk’s title is a useful question to keep in mind.

1️⃣ A large prompt: find out whether the model can solve the task The original pain was mundane: colleagues kept interrupting analysts with questions about customers, products, and metrics. Qu started by putting a Snowflake schema into the system prompt. The model generated SQL; he copied and ran it manually. That was a useful, inexpensive experiment. But there is still a gap between executable SQL and a correct answer to a business question: you must choose the right metric, period, filters, and joins. The next version had to carry out the whole process.

2️⃣ A chain of specialists, then one agent with a shared history The team split the work into stages: explore the schema, plan the query, execute SQL, and prepare a report. Each agent had a narrow role and its own tools. According to Qu, the trouble was the context handoff: the next participant received only a brief summary of the previous step. When execution exposed a bad assumption, the system needed to return to exploration. So the stages were combined into one agent that retained its working history and could switch among planning, execution, and checking. It would be premature to generalize this result to every multi-agent system: the specific problems here were a rigid sequence and context loss during handoff. Early users still rated the new system poorly, because their questions fell outside the scenarios its builders had anticipated.

3️⃣ A working directory instead of ever-growing orchestration The next turn came when, Qu says, Claude Code with Opus 4.5 began answering their questions noticeably better than their own agent. The team brought that approach into d0: an isolated environment, semantic-layer files, ordinary tools for reading and searching, bash, and operations specific to Vercel’s tasks. The semantic layer describes metrics and relationships among entities. The agent could now explore those descriptions and return for more detail when needed. Qu says its score on internal evaluations roughly doubled.

This immediately raises two questions: ❔ How much of the improvement came from the architecture, and how much from a stronger model? The talk does not isolate those effects. ❔ How convincing are the tests themselves? The “100% success rate” in a related article means five successful queries out of five, up from four. It should not be combined with the doubled score from the talk: the comparability of the test sets is not established.

Incidentally, Qu notes in the article that the team already had well-documented data. The file system gave the model access to existing knowledge. Anyone transferring this solution to another company would need to assess the quality of that knowledge separately.

4️⃣ Recurring requests become skills As d0’s use grew, the team noticed that many requests followed similar patterns. According to Qu, a recurring job analyzes recent requests and extracts skills from them; there were about a hundred by the time of the talk. The next run can use an established procedure for a familiar task. In eve, these skills are instructions loaded when needed.

This is where things get especially interesting. Who checks whether a popular way of calculating a metric is also correct? How is a skill updated when business logic changes? How do you roll back a procedure that spreads one error into new answers? The mechanism for generating skills is described, but their review, updates, and removal remain offstage. Repetition alone does not guarantee quality.

5️⃣ A useful internal experience becomes a framework This path produced eve: instructions, skills, tools, and communication channels are defined in files, while the framework assembles an agent from them and provides the execution environment. Qu reports thousands of requests to d0 each day and roughly 20 agents in active use within Vercel. Those are signs of adoption. To assess the effect, I would also want to see the share of correct answers, the time spent verifying them, and the cost of maintaining the system. The claim that analysts were freed up for more substantive work sounds plausible, but the talk does not provide measurements of time saved.

The journey of d0 shows how building an agent gradually reaches into the structure of a company’s own knowledge. The most interesting follow-up talk would be about the lives of those hundred skills: who is responsible for their correctness, and how does the team recognize when accumulated experience needs to be reconsidered?

#AI #AI4SDLC #Agents #Data #Architecture #Evals