[5/7] Meta's Hyperscale Infrastructure: Overview and Insights (Category: Infrastructure)
[5/7] Meta's Hyperscale Infrastructure: Overview and Insights (Rubric #Infrastructure)
In this post, we will continue to consider a cool article from the banned in Russia company Meta. (previous issues: 1, 2, 3 and 4) And we'll talk about reducing the cost of infra.
All global datacenters as a computer Meta’s approach to the perception of its infra is perfectly described by another insight
Insight 6 : Meta is evolving from the practice of “the datacenter as a computer” to the vision of “all global datacenters as a computer.” In this model, the infrastructure autonomously determines and migrates deployments across global datacenters in response to workload changes, eliminating the need for user involvement. We have successfully demonstrated this approach for databases, ML systems, and diverse services operating at the scale of O(100,000) servers and O(100,000) GPUs.
Hardware and software co-design But at the same time, there is a need for joint design of software and iron for it. - Graceful degradationEfficient infrastructure must be able to adaptively degrade in extreme situations, so as not to keep a constant excess supply of power in reserve. The Meta system Defcon Disables functionality by priority levels, freeing up resources for key services Savings on proxy in service meshService mesh architecture with sidecar proxy per service that intercepts and routes requests is common in the industry. Meta developed its own ServiceRouter system (~1The percentage of RPC requests go through proxy. 99% – routed directly using the library built into each service). This saves 100k+ servers. - Multilevel data storageIn order to optimize storage costs, data is divided into categories by frequency of access and allowable delay.
- Hot data. (Sotsgraf, ribbons, caches) stored in high-performance systems (RAM + SSD)
- Warm data. (Photo/video of users, clickstream) stored in the Tectonic distributed file system on conventional HDD disks (1 server ~36 HDD + 2SSD for metadata)
- Cold data. (High quality original video) archived on ultra-dense storage servers with a large number of slow disks (1 server ~216 HDD) Local SSDs instead of network storageIn the cloud services industry, it is considered good practice to take storage separately to a block device for ease of migration and load balancing. But for the sake of savings and low latency, Meta prefers local SSDs even for stateful services where possible. This creates complexities that Meta solves centrally through the sharding control system. n (Shard Manager), which abstracts the placement of data fragments and provides automatic rebalancing Cheap equipment with reliability through softwareIn public clouds, hardware is often duplicated because customer applications may not be ready to crash. Meta has taken the opposite approach – using simpler and cheaper hardware, but making everything fail-safe. At the end of the day, the next story sounds like this.
Insight 7 : To reduce hardware costs, we use software solutions to overcome the limitations of lower-cost hardware. Although this approach adds complexity to the software stack, we consider the trade-off worthwhile due to the significant cost savings.
In-house hardware design For everything described above, Meta itself develops data center designs. (Open Compute datacenters)and a large part of the equipment. Control over design allows you to remove all unnecessary and increase efficiency (especially energy efficiency, which is now a bottleneck for DC)
Insight 8 : To reduce hardware costs and power consumption, Meta designs its own datacenters, servers, racks, and network switches, and shares these designs through open source.
In next post We'll talk about how engineers at Meta design their systems.
#Infrastructure #PlatformEngineering #Architecture #DistributedSystems #SystemDesign #Engineering #Software #DevEx #DevOps