Cultivating Production Excellence • Liz Fong-Jones • YOW! 2019

Another one. report About SRE practices from an engineer who got them working at Google. Liz Fong-Jones only in the beginning 2019 I went to Honeycomb a year ago. 10 I worked at Google for years. Interestingly, the new company Liz is engaged in observability tools that are useful to engineers who care about the reliability of their systems.

The main points of the report are such Production systems are getting more complex. (Especially when using microservices and big data) Reducing this complexity can be difficult. What is reliability and how to measure it (And what is uptime?)? Do not buy DevOps – these are not boxes like Honeycomb, IaaS, K8s, and other cool words. Devops is a culture.

• Further discussion of allerts, dashboards, predictability of depletions, etc. The return from technical systems to sociotechnical systems and the importance of thinking about the people who develop and maintain our systems Liz suggests investing in culture, people and processes to improve the reliability of systems – and that’s what she calls production excellence. To improve production excellence, it offers a range of measures. Make a plan, stop with metrics and watch them improve.
• Involve everyone. (Not only engineers, but also products, financiers and backpackers.) Liz proposes to start by teaching you to determine that something is wrong with the sale and be able to fix this problem. And the more difficult advice is to eliminate unnecessary complexity. (but easier said than done):) If systems are constantly broken for certain reasons, it is worth eliminating these reasons. For measurements, Liz suggests using already standard SLI/SLO/SLA indicators, goals and agreements on the level of service provision. (It should be used as a common language with non-engineering specialties: products, financiers, etc.) Next, Liz explains that to measure these metrics, you need to understand which critical user journeys to continue thinking about specific events. (events) You need to be able to identify a good event or a bad one. (and record both successful and unsuccessful events - from here you can count availabilty) Expose thresholds for availability Chaos experiments to determine the sensitivity of users to the level of service are also mentioned. Next, we are talking about how to understand which window to use to calculate indicators (day, week, month, etc.) Importance of not putting too much emphasis on reliability – it is important to find a balance between the cost of reliability and the ability to allocate time to the development of the service through the addition of new features How to use SLO for alerts and error budgets How to use data to understand how much time a team needs to spend on reliability (Advice to budget for errors) In addition to SLI & SLO, you need to be able to solve problems on the market:) You need to use observability tools. (By the way, we have our own observability platform Sage at Tinkoff, which you can even touch outside.) Next, Liz says that we need to be able to distinguish why we have deviations in the performance of different services. It ends with a return to cultural issues: Heroism is not a sustainable strategy for solving problems. Debugging is a collaboration, you have to train teams to work together. Document architectural decisions and how our systems work Use common tools and platforms Make blameless postmortems for incidents Plus, at the end, it’s about risk management – about probability and impact, which determine the level of risk. You can usually reduce the likelihood of problems by selecting those that affect SLOs and prioritizing those tasks in the backlog. We need to start by improving observability.

P.S. On the subject of reliability, you can read the materials from my post.Designing reliable systems"

#Software #Engineering #Architecture #SoftwareArchitecture #SystemDesign #DistributedSystems #SRE #Reliability #Conference