Scaling compliance control of reliability on hundreds of services - Salikh Fakhrutdinov, T-Bank

Interesting. report My colleague, Salih, is a Staff SRE engineer at Origination Business Platform. In this report, Salih talked about how you can ensure the reliability of hundreds of different services by controlling invariants, which is implemented through a third-party application. This app configures different rules for checks, such as using different ports for samples. (liveness, readiness, startup), settings java applications (fall), DBMS parameters, presence of allerts for critical SLA and so on. These rules can be updated reactively. (after-incident) or proactively based on research to improve application reliability. This application collects information about compliance with the rules and allows you to create a report in the form of a leaderboard. Those managers whose services are at the bottom of the leaderboard, usually have a great motivation to fix the problems found.

In principle, another implementation of this logic of checks is the implementation inside the Pipeline delivery, about how our Quality Gate works, which checks the level of automation testing. But the guys decided to go the other way to be able to check the prolapses not only before the deploitation or on time of deploitation, but also further as they work in production.

P.S. In software architecture, such checks are offered through fitness functions, which were proposed in the book Building evolutionary architecture. Here. podcast "Code of Architecture" where we talked about this concept.

#Architecture #Management #Leadership #SRE #Engineering #Software