[2/2] A Field Guide to Reliability Engineering at Zalando • Heinrich Hartmann • GOTO 2024 (Category Management)

Continue. story speech Heinrich HartmannHead of Reliability Engineering at Zalando SE, I would like to tell you about the other approaches and principles that the guys practice.

9. Many infra-reports are stitched into the platform and teams do not need to collect them with their hands. K8s, redis, jvm and other dashboards exist out of the box, and there are also guidelines for making good reports. The author tells us about RED, 4 Golden Signalswhich they track
10. Next, the author talks about observability, which allows you to answer complex questions about your system. The guys use OpenTelemetry to standardize app signals, and they also have distributing tracing. More about observability, I recommend looking at it. 2 Talks I mentioned earlier:

• What Is This OpenTelemetry Thing? • Martin Thwaites • GOTO 2024
• Observability in an Asynchronous World • James Eastham • GOTO 2024

11. In Zalando, guys use SLO. (service level objectives) to work with reliability and this allows Understand the reliability of users Manage engineering investments in reliability Assess the impact of incidents Tune the rules of allerting in the direction of improving their quality The guys have reports that are audited by management regarding the SLO compliance with various business operations. In general, this leads to another rule: Rule of operations #4: SLIs quantify the reliabilty of a user experience
12. The guys have a process for dealing with incidents and even have a fifth rule: Rule of operations #5: Past failures lead the way towards future reliability. In fact, the guys write and review postmortems, dig into root causes, and also put and most importantly polish tasks in order to prevent these problems from recurring.
13. Finally, the author leaves a story about how the management of all this economy looks and formulates a rule. Rule of operations #6: You get what you inspect. In fact, he talks about weekly meetings with directors and reviews of reliability reports.

The author concludes by listing all the principles in one list. (For every item I'm willing to sign up for.) 1. Obsess about user experience 2. Engineering for reliability involves people as much as involves technology 3. Alert on user experience ("symptoms") not on server experience 4. SLIs quantify the reliabilty of a user experience 5. Past failures lead the way towards future reliability 6. Past failures lead the way towards future reliability

All in all, this is a very good report on Zalando’s approach to reliability.

P.S. The author of the report has a great personal blogThere is a lot of interesting material.

#SRE #Architecture #DistributedSystems #Software #Observability