turbopuffer: How to Build a Search Database on Top of S3 (Category Architecture)

I watched Gergely Orosz’s conversation with Simon Eskildsen, co-founder and CEO of turbopuffer. Formally, the episode is about search infrastructure for AI products, but to me it is primarily about an old engineering discipline: calculate the physics and economics of the system first, and only then trust benchmarks. Before turbopuffer, Simon spent eight years working on Shopify’s infrastructure. There he created Napkin Math, a table of DRAM and NVMe throughput, S3 latencies, and the cost of different storage types. If a calculation says “10 ms” while a test shows 10 seconds, the answer is not to choose a different database, but to understand where three orders of magnitude disappeared: in the query plan, the network, distribution of work across nodes, or the experiment itself.

turbopuffer grew out of exactly this kind of discrepancy. While building recommendations for Readwise, Simon estimated that storing and searching vectors would cost about $30,000 per month, versus $5,000 for all the company’s other infrastructure. According to him, the product economics did not work, so the feature was not launched. He then asked a more useful question: do all vectors really have to remain permanently in expensive memory and on replicated SSDs?

The answer was a database in which all durable data lives in object storage, while compute nodes hold no state of their own. In simplified terms, vectors are grouped into clusters, an index of centroids is stored separately, and a query loads only the nearest clusters. Hot data goes into memory, warm data into the NVMe cache, and cold data is read from S3.

This is not a free trick: a write can take up to 200 ms, while cold queries sometimes run into hundreds of milliseconds. But search usually values high throughput, reliability, and storage cost more than the transactional latency of every write. turbopuffer deliberately pays with slower writes and occasional cold queries in exchange for cheap storage of the bulk of its data.

The story of the first customer is especially good. According to Simon, it was Cursor, which approached them after the MVP launched on Twitter. Simon flew to San Francisco and started not with a sales pitch, but by diagnosing someone else’s Postgres problem: autovacuum could not keep up, and the database was reading the table instead of using the required index scan. That help created trust; Cursor then moved its workload to turbopuffer in one or two weeks. According to the company, the first bill was 95% lower than the final bill from the previous provider. This is not an independent benchmark, but the product lesson is strong: critical infrastructure is bought not from the team with the prettiest diagram, but from one that understands the entire failure and cost envelope.

My main takeaway is that first-principles thinking does not replace measurement—it makes measurement meaningful. First estimate the hardware limits and workload cost, then build the simplest system with the necessary invariants, and only after that optimize real work. turbopuffer is interesting not because “S3 defeated databases,” but because the team refused to pay for properties its search did not need.

#Software #Data #Architecture #Infrastructure #Engineering #AI