[6/7] Meta's Hyperscale Infrastructure: Overview and Insights (Category: Infrastructure)

[6/7] Meta's Hyperscale Infrastructure: Overview and Insights (Rubric #Infrastructure)

In this post, we will continue to consider a cool article from the banned in Russia company Meta. (previous issues: 1, 2, 3, 4 and 5) And we're going to talk about how kids approach scalable application design.

Centralization vs decentralization Planetary-scale infrastructure has historically been associated with decentralized architectures. (BGP, BitTorrent, etc.). They scale well without SPOF. (single point of failure). However, Meta’s experience has shown that within a data center, where resources are relatively reliable and managed by a single organization, centralized controllers often simplify the system and provide sufficient scalability. And often it also allows you to make more globally optimal decisions than many local agents. So Meta has deliberately moved away from many originally distributed designs to centrally managed designs. For example,

• Internal data center network (Fabric) It still uses the protocol BGP for interoperability, but routing is controlled by a central controller, which, when the links are overloaded or broken, re-optimizes the traffic paths instead of the slow converging dynamics of BGP
• On the global backbone (WAN) Meta Initially, it used a decentralized lane reservation protocol. (RSVP-TE), but then switched to a central controller, calculating the optimal paths for flows between data centers and laying backup channels in advance in case of typical failures. This made it possible to use channel bandwidth much more efficiently and simplified network management.

In general, the Meta approach can be formulated by such an insight.

Insight 9 : In a datacenter environment, we prefer centralized controllers over decentralized ones due to their simplicity and ability to make higher-quality decisions. In many cases, a hybrid approach - a centralized control plane combined with a decentralized data plane-provides the best of both worlds.

As an example, a hybrid service mesh called ServiceRouter (Trying to get the “best of two worlds”). ServiceRouter handles billions of calls per second between microservices distributed across millions of L7-level software routers. In traditional solutions, service mesh (for example, Istio) Each application is accompanied by a local proxy through which all outgoing and incoming calls pass. ServiceRouter Meta rejected this scheme (as mentioned, ~99Percentage of requests are made without sidecar proxy). Instead, - Control plane centralized. It aggregates all information about services and global network metrics, calculates optimal routing rules and stores them in the RIB. (outing Information Base)built on top of a distributed database Delos on Paxos protocol (That is, it is distributed and resilient.). Thus, the central ServiceRouter controllers are only responsible for computing global solutions, and the direct routing work lies with the data plane. Data plane as separate L7 routers decentralized They automatically pull information from the RIB. (cache a small necessary subset) and operate autonomously, without the constant involvement of a central coordinator

Thanks to this design, they are achieved. Simplicity of management - the whole picture is centrally visible Scalability – There is no bottleneck through which all traffic passes As a result, it is possible to provide full functionality of the service mesh. (balancing, retries, discovery, monitoring) with minimal resource consumption and with the possibility of global optimal load distribution.

In last post From the series, we will talk about the future directions of Meta infrastructure and architecture development. (This is one of the most interesting parts.)

#Infrastructure #PlatformEngineering #Architecture #DistributedSystems #SystemDesign #Engineering #Software #DevEx #DevOps