Reliability at scale 50 Millions of customers: T-Bank experience for engineers (Category SRE)

Habre's out today. article Alexei Merson, former dev advocate of our platform Sage. Lesha talked about how we have built reliability processes and how they are instrumentalized. Below I briefly described the main thoughts of the article.

Lesha started with the database, telling the definition of reliability by SRE Book from Google, the pyramid of quality. (reliability> UX -> Enjoyment)Without a solid foundation, there is no point in talking about user-friendliness. Then came the three: SLI/SLO/SLA, as well as the error budget and its connection with the canonical MTTR and MTBF. Then Lesha told about the holy trinity of instruments. -- Sage A single observation platform that replaced the zoo with monitoring tools -- FineDog Incident management system to account for SLA and health services Kolobok - a release management tool with a calendar and traffic light But tools usually support processes, so Lesha talked about dealing with incidents: from stateless engineer alerting to crisis management plans, and also explained how we unload the contact center during incidents. Reliability issues are overseen by our reliability center, which works on methodology, processes, makes a deep dive on major incidents and shares with all statistics and interesting insights.

If you sum up and somehow summarize what is useful to do in a company of almost any size right now, you get the following: Observation should be done regardless of the size of the service. It is important to work with incidents and learn how to write postmortems. If the company is large, a hybrid scheme with a reliability center and chief reliability officers roles on individual business verticals may be required. It is important to focus on customer experience – technology is made to solve customer problems If you look at scale T, you can see that Automation of all processes is important - you can not personally talk to millions of customers. It is important to collect and analyze large amounts of data. It is important to design fault-tolerant systems and use platform services (XaaS)

If we talk about the value of the article, it not only analyzes the case T in terms of reliability, but also demonstrates an integrated approach to reliability: from technical tools to organizational processes. This is not just a story about how we monitor metrics, but a systematic vision of how to ensure the performance of a critical service.

P.S. I wonder if I've been lately. performer report**Are reliability and security additional options or the foundation for modern IT systems?**On PHDays, he mentioned the processes and tools that Lesh discusses in detail in this article.

#SRE #SystemDesign #Software #Architecture #Metrics #SoftwareArchitecture #Engineering