[1/2] A Field Guide to Reliability Engineering at Zalando • Heinrich Hartmann • GOTO 2024 (Category Management)

I recently saw something interesting. speech from Heinrich HartmannHead of Reliability Engineering at Zalando SE, about their approaches to reliability.

The main message of the report is as follows:

1. Heinrich notes that he talks about the experience of Zalando, a company that is not big tech and can not afford the sweep of Google in terms of SRE. Therefore, the purpose of the presentation is to share knowledge and experience for small and medium-sized companies.
2. The guys in Zalando have a clearly articulated mission for SRE.

Mission: Protect the User Experience from operational failures while keeping an eye on Developer Productivity and On-Call Health The following is the Zalando rule of operations: 3) Rule of operations #1: Obsess about user experience The user experience should strive for perfection, but there are productivity metrics and on-call health metrics that are compensatory when trying to improve reliability. It’s easy to be reliable if you don’t change your system. (This reduces productivity.) or burn people on duty (It reduces on-call health). The SRE Triangle: Reliability - Productivity - On-Call Health. Interestingly, according to our reports within the framework of the framework SPACE In T-Bank, you can see that the lowest satisfaction is with SRE engineers. (I was talking about the SPACE framework. here). 4) Rule of operations #2: Engineering for reliability involves people as much as involves technology In fact, reliability is not only about technology, but also about people. And the bigger the company, the more it is about people and processes:) So we have to look at this from a sociotechnical perspective. The author recommends studying systems thinking for this and recommends the book "The ABC of Systems Thinking" ("Thinking in Systems: A Primer")Which I'm talking about. told. Systems thinking will allow you to learn to take into account secondary and tertiary effects by taking into account feedback loops. Interestingly, feedback loops are part of another framework for developer productivity called DevEx. told earlier 5) Next, the author recalls conway and mantraYou build it you run itas guidelines for the organization of technological and organizational structure 6) Author: system model of Zalandowhich he uses in his thoughts:

• Management upstairs. (25 director)
• Engineering in the middle (250 team 3.5k of annexes)
• Platform at the base (k8s, Postgres, Kafka, CI/CD, Observability, ..., 20 team) As a result, the guys are trying to carry the complex tasks into the capabilities of the platform and take the load off the teams. Narpimer, scalability and resource planning are integrated into the platform, or the ability to perform load tests.

7. Further, the author notes what Zalando is good at: backend and microservices infrastructure, as well as the observability of this, and what you want to improve: understanding the user experience, as well as the reliability of data systems and business processors. 8.) Next, the author tells about the alerting and shares another operation rule: Rule of operations #3: Alert on user experience ("symptoms") not on server experience ("causes"). Technical directors review duty reports on their teams to track the workload on engineers. If the load is too large, then you need to deal with it.

Continuation of the review in next post.

#SRE #Architecture #DistributedSystems #Software #Observability