[1/2] How AWS S3 is built (Category Architecture)
Interesting. podcasting "The Pragmatic Engineer", in which Gergely Orosz communicates with Mai-Lan Tomsen Bukovec, VP of Data & Analytics at AWS, which leads the development/operation of S3. Simple Storage Service is one of the largest systems in the world, so you can learn a lot from this discussion.
1The scale that breaks intuition In S3 hundreds of millions of transactions per second, 500+ trillion objects, hundreds of exabytes of data. Interestingly, Gergely and Mai-Lan say that stacked tens of millions of disks in S3 will almost be able to reach the ISS.
2Switching to Strong Consistency – Like a Cool Engineering Migration S3 launches with eventual consistency (2006)But then moved on to strong consistency without compromising availability or cost to customers. The architecture was something like this: replicated journal + cache coherence protocol with the idea of failure allowance.
3Quiet transition to Rust in critical request path The team rewrote almost all performance-critical in processing requests for Rust, with motivation: maximum perf and minimum latency.
4️⃣ 11 Durability is a measurement of fact, not a promise. Durability level 99.999999999% is supported not by magic, but by a fleet of background services: micro-audit services that continuously check each byte, and individual repair mechanisms that are automatically repaired.
5Formal methods are production practices, not academic delights S3 actively uses formal verification methods: when changes in the index/consistency subsystem, automatic formal checks are launched so that there is no regression of the model. And it's not just words: Amazon Science: Lightweight formal methods Experience where such methods helped not to put on sale 16 problem
6The main enemy today - correlated failures Single breakdowns are normal, but correlated failures are dangerous: general rack/AZ/power/etc. The architecture is built around fighting these correlations: AZ replication, quorum-approaches, physical/logical decorrelation, copy storage in different fault domains.
7Hundreds of microservices - and many of them are not about traffic The episode features order. 200+ services, and a significant part of them deals with health checks / audit / repair, not user requests. Difficulty is maintained through simplification – each service should be focused as much as possible.
8S3 stops operating only with bags: new primitives Tables and Vectors There are new primitives.
- S3 Tables: object storage with built-in Apache Iceberg support and background table optimization (repackage/compact etc. "in the background"). AWS claims up to 10x TPS vs Iceberg Tables in conventional S3 Buckets
- S3 Vectors: native storage/search of vectors in S3. In the episode - the engineering idea of vector neighborhoods (offline)to get warm queries <100ms, and very large index/bucket scales
9Crash consistency as a worldview Real engineers think that the system must return to the correct state after fail-stop; design goes through the search of possible states in case of failure + a set of services that hold the invariants.
Scale must be to your advantage You can’t build in such a way that the growth of service degrades performance; on the scale, on the contrary, there should be effects that improve reliability. (for example, load decorrelation)
In continuation I will tell you how to transfer this experience to practical recommendations to engineers and technical managers.
#Culture #Management #Leadership #Processes #Engineering #Software #Architecture #DistributedSystems #SystemDesign