Taxi reliability tools \ Alexander Fisher, Yandex Taxi

Taxi reliability tools Alexander Fischer, Yandex Taxi

Interesting. speech Alexander Fischer on processes and tools to improve reliability The presentation is short and consists of six parts. 1. Taxi and reliability Introductory about the criticality of taxi services, the load profile, briefly about the architecture, the technologies used. More about reliability metrics:

• Standard: SLO in nines, MTTR (mean time to recovery)
• Interesting: MTTRC (mean time to root cause) How fast is the root cause of the failure, ROCOF (rate of occurrences of failures) - failure rate In general, the message is that in the following paragraphs the author talks about the tools to combat complexity.

2. Chaos. The approach is that they use fault injection, where the data plane of this chaos is implemented through lua scripts for nginx, which balance traffic, and there is a control plane to control the configuration of these injectable errors. This tool allows

• Slow down service. (but within the propogation deadline) To simulate the metastable state of the service, when the service fell under load and cannot stand up, he is waiting for someone to help him:)
• Extradition 500 percentage errors 3. Load reduction: degradation, retries, limits Guys have. Gracefull degradation mode, where non-core functionality can be disabled (adds 20Percentage of power, but in plans 50%) Manage retries to solve the problem of amplification of requests during crashes There is a classification of services by levels, where the level depends on the criticality of the service for the ability to perform the main function. (taxi off) 4. Virtual orders The story is about testing the load on the system through simulation of the load with virtual orders, which allow you to check the load not of the endpoint API, but Holistically for the entire system. A cool tool that solves the problem of checking nonlinear dependencies and unpredictable behavior of a really complex system depending on different types of load. 5. Observability and eventboard Here the author shows and talks about the dashboards for observability + interesting eventboard about the events on the sale, as well as the button “roll back everything in the last n minutes”, which helps to reduce MTTR. (mean time to recover) and speed recovery after failure.

6. Simulation of incidents Here's a story about the crash coordinators + about the malfunction simulation and the troubleshooting training that goes on every week.

At the end of the talk, the author talks about experimental instruments. Autorecovery is a robot that automatically repairs incidents. (He's still in dry run running and running.) SRE GPT - a tool to help find root cause

In general, I liked the report: an interesting story, funny thoughts, quite practical recommendations that can be used in your processes of improving reliability.

#SRE #DistributedSystems #Reliability #Architecture #Software #Processes #Management