[4/7] Meta's Hyperscale Infrastructure: Overview and Insights (Category: Infrastructure)

[4/7] Meta's Hyperscale Infrastructure: Overview and Insights (Rubric #Infrastructure)

In this post, we will continue to consider a cool article from the banned in Russia company Meta. (previous issues: 1, 2 and 3). We’re going to talk about how developers within Meta can be more productive.

Continuous deployment and automation One of the main goals of Meta’s overall infrastructure is to accelerate the work of developers. The company brought to extreme CI / CD approaches, achieving almost complete automatic release of updates. 97% of Meta services are deployed without manual participation of engineers - changes are delivered through automatic piplins of the Deploy (More than 30k Pipelines are monitoring updates). Around 55% of services use a really continuous depot, the rest42% - Deployed by robots on schedule (usually daily or weekly). For example, the front-end platform Meta (serverless functions that serve user requests) It is released every three hours, and it runs on 500k+ servers and its code is changed daily by 10k+ developers.

Configuration as code and instant changes In Meta, the distinction between “code” and “settings” is virtually erased—configuration changes are handled by the same pipelines as the program code. Every day, more than 100k configuration changes are automatically applied to production using an internal settings management system. They affect about 10k different services and 1M+ running processes around the world. (settings of load balancing parameters, inclusion of feature flags, settings of A/B tests,). Almost every Meta engineer who writes code also makes edits to live configs:

• They're stored in turnips as code.
• Take a peer review.
• Running through CD pipelines Agents on the principle of publish-subscribe roll out changes to services Applications apply new parameters on the fly, without restarting processes From this follows the fourth insight of the article

Insight 4 : Even for a large organization with O(10,000) services, it is feasible to adopt continuous deployment at extreme scales and speeds. Specifically, 97% of our services adopt fully automated deployments without manual intervention, and 55% deploy every code change instantly.

Tools for quality and quick rollback The desire to release immediately inevitably increases the risk of failure, so Meta has developed multi-layered tools for safe deployment. Before full rollout, the new code undergoes automatic tests and canary runs. If problems are detected, the mechanisms of instant rollback to the previous stable version are well established.

Serverless functions as the basis of development More 10 000 Meta developers use FaaS on a daily basis, eliminating the need for infrastructure management: code automatically scales and deploys and makes optimal use of the infrastructure. The use of FaaS is integrated into the IDE (facilitated access to the social graph and backend system n). FaaS is a stateless architecture that relies on external cache systems and databases to provide predictable behavior and ease of horizontal scaling. Meta has two FaaS platforms:

• FrontFaaS for processing user requests (PHP, Python, Erlang, Haskell) low latency
• XFaaS for processing asynchronous, event functions with sharp peak loads. They are optimized through global balancer, deferred execution, and quota-trottling to avoid overprovision. This part summarizes the fifth insight.

Insight 5 : Serverless functions have become the primary coding paradigm for product development at Meta. More than 10,000 Meta engineers write code for serverless functions, exceeding the number of engineers writing regular service code by 50%. In next post We're going to talk about how Meta reduces its infrastructure costs.

#Infrastructure #PlatformEngineering #Architecture #DistributedSystems #SystemDesign #Engineering #Software #DevEx #DevOps