Mixedbread: How to Organize Knowledge Work for an Agent (Series #AI)
Finding “30 days” in a contract is easy. Understanding what the period means, whom it applies to, and which conditions change it is a separate task. In Benjamin Clavié's Mixedbread talk, this example neatly explains why successful code search does not yet mean an agent is ready to investigate an arbitrary document archive.
His thesis: agents doing knowledge work should be designed around how research itself is organized. Code has function names, paths, and explicit relationships. Developers also often turn a problem into a specific ticket before handing it over. In legal or analytical research, the agent has to work out which questions to ask in the first place.
Clavié suggests looking at a law firm. A partner understands the client’s situation, identifies research directions, and delegates the searches to assistants. They return short memos, which the partner uses to assemble a solution and request clarification. The corresponding agent workflow looks like this: define the task → investigate individual questions → return research memos → produce the overall answer.
This brings together two themes of the talk. A good tool makes retrieval more affordable in time and cost. Dividing the work helps make use of the findings without dumping the entire archive into one agent’s context. In Clavié’s view, tools and work organization need to evolve together.
There is a concrete product side to this argument, too. Mixedbread builds retrieval infrastructure for agents, including processing for PDFs, images, audio, and video. The company develops its own Wholembed v3 model and Silo search engine.
One detail in its technical write-up is particularly interesting: content is represented by multiple vectors that retain information about its individual parts. This approach, late interaction, is intended to better distinguish documents on similar topics that differ in consequential details. PDF pages are indexed as images to preserve tables and layout, while text is extracted alongside them for reading. That precision comes with storage and computation costs, which helps explain the custom engine.
Toast 1 is a specialized search agent. It breaks a question into subqueries, searches, reads sources, and assembles context for the main model. It is the assistant from the law-firm example, with an API.
The company has a telling Harvey LAB evaluation. On a subset of 33 tasks, the same GPT-5.6 Sol agent consumed 80.6 million tokens with filesystem search, 47 million with Mixedbread Search, and 23 million with Toast 1 on top of that search. The answer-quality score stayed the same across all three configurations. This is Mixedbread’s own measurement on a small subset, but it illustrates the economics of dividing the work: a comparable result with roughly 3.5 times fewer tokens. These figures come from the company’s materials, separately from the talk.
I would push back on a sharp divide between programming and other knowledge work. Software development also has plenty of tasks where the first step is figuring out what we are building. The degree of uncertainty in the task definition matters here. The research memos, though, deserve a closer look. I would expect such an assistant to return specific sources, the conditions under which its conclusions apply, and a list of unresolved contradictions. Otherwise, the main agent has to trust someone else's retelling—a familiar problem when delegating work to people.
#AI #Agents #Architecture #Engineering #Evals