An Illustrated Guide to AI Agents: The Complete Edition (#Agents column)

Back in March, I wrote about "An Illustrated Guide to AI Agents". At the time, it was an Early Release: six chapters were ready, and I promised to keep an eye on what came next. O'Reilly has now published the complete first edition — 450 pages and 10 chapters. The reason to return to it is not merely the extra pages. What once looked like a strong half of a future textbook has become a coherent route from the internals of an LLM to evaluating and operating an agentic system.

Compared with the table of contents available in the spring, four major chapters have arrived.

1️⃣ Large Language Models makes the book self-contained. It covers tokens, system prompts, tool calls, pre-training and post-training, the Transformer, context length, the KV cache, and Mixture of Experts, consistently tying them to agent behavior, latency, and cost. It becomes easier to see how the model itself, context length, and the KV cache affect the system as a whole.

2️⃣ Evaluating Agents is arguably the most important new chapter. The authors cover public benchmarks, outcome and trajectory evaluation, LLM-as-a-judge, rubrics, reliability, safety, and custom evals. One especially useful distinction is between two metrics: pass@k asks whether an agent succeeds at least once in k attempts, while pass^k asks whether all k attempts succeed. The former can be enough for an impressive demo; a real workflow depends much more on the latter.

3️⃣ Multi-Modal Understanding explains how an agent gains the ability to see images, hear audio, and process video: an encoder turns the input into embeddings, a connector maps them into the format expected by the LLM, and the model uses them as context. The trade-offs are made clear as well: projection is simpler and cheaper, query-based connectors compress long inputs more effectively, and fusion is often more expressive but also more complex and expensive.

4️⃣ Code Agents and Code LLMs takes the story from generating a function to working inside an entire repository. It covers file and terminal tools, SQL, code sandboxes, repository maps, context caching and compaction, planning, tests, and human approval. One important point is that a fixed workflow is sometimes more reliable than a full agent, while shell access is not a convenience toggle but a security boundary that determines the potential blast radius.

The final version of the already familiar Multi-Agent Systems chapter also includes orchestration patterns, communication protocols, deep research agents, and scenarios such as an AI co-scientist. Yet the book's coherence comes from its through-line. Part I incrementally assembles a single TinyAgent: LLM → reasoning → memory → tools → planning → evals. Part II shows how that same architecture specializes into multi-agent, multi-modal, and coding systems. The result is no longer a catalogue of fashionable frameworks but a comprehensible engineering cycle.

For engineers, this closes several concrete scenarios:

• designing an internal research or support agent and deciding where context belongs and which tools it needs;
• putting prompt, model, and memory changes behind regression evals that inspect both the answer and the trajectory;
• building a code agent with repository search, tests, and a sandbox;
• adding images, calls, or video where a text-only agent is blind to part of the task.

Tech leads and executives face a different set of scenarios:

• deciding where an agent is justified and where a deterministic workflow is enough;
• choosing between single-agent and multi-agent architectures while accounting for coordination cost;
• comparing vendors on the organization's own tasks as a “model + harness” pair, including reliability, latency, and cost;
• setting autonomy boundaries around permissions, irreversible actions, safety cases, and mandatory human approval.

If you read the early chapters in the spring, there is no need to start over: skim Chapter 2 for the link to agent economics, do not skip Chapter 7, and then continue with the multi-modal and code-agent chapters. That sequence is what turns the early collection of strong explanations into a complete engineering book.

#Agents #Books #AI #Engineering #Architecture #Management #Software