The evolution of SRE: the introduction of an AI assistant in T-Bank (Category SRE)
I saw something interesting. speech Ivan Yurchenko, head of FineDog Growth in T-Bank, which is engaged in the development FineDog incident management platformIt helps the bank to quickly identify service failures and reduce the time for their elimination. This report was presented at the conference Platform Engineering Night And it's about introducing AI assistants into the work of reliability engineers.
By the way, I already talked about three reports from Platform Engineering Night:
- "AI and Platform Engineeringfrom Igor Maslov (VP T-Bank at T-Bank)
- "Developing your own AI assistant for code: sprint or marathon?from Denis Artyushin (tech product & engineer at T-Bank)
- "AI assistants when working with code. Looking to the futureby Evgeny Kolesnikov (tech product at Yandex)
I will tell you about the main ideas of Ivan’s report.
1. The evolution of incident management tools Ivan presented the concept of three-level development of tools for dealing with incidents Platformization – combining disparate tools into a single ecosystem to better manage incidents Automation – simplification of routine actions, which reduces the response time to incidents Robotics – Automation of decision making using AI 2. The Incident Life Cycle and the Role of AI Ivan described in detail the stages of dealing with the incident: detection, handling the incident and post-analysis. Emphasis was placed on the importance of collecting context for effective problem solving. AI assistant is now integrated into all stages of this cycle, helping not only to respond to incidents, but also to predict them, analyze logs and suggest solutions before a critical situation begins. 3. Innovative projects based on AI The main projects about which Ivan told the following Summarizer is a system for information aggregation and automatic generation of post-analysis incidents. Uses AI to improve analytics and identify patterns in incidents LogAnalyzer is an advanced tool for analyzing logs and finding related incidents. Uses log clustering to isolate anomalies. It looks something like this.
- Logs are downloading from Sage (platform observability) every 5 minute Next, the text is pre-processed and segmented into fragments. Then vectoring with TF-IDF and transformers In the final, data is visualized in three-dimensional space to visualize anomalies. 4. SRE Assistant and its capabilities Ivan introduced an AI assistant designed to support SRE teams, which has the following capabilities: Integration with Time corporate messenger for easy access Interaction with various bots and LLM agents
- Performs key scenarios: Incident management and post-analysis generation Receiving information from the database Searching the knowledge base using the RAG platform
- Management of duties through the messenger 5. Metrics and evaluation of the effectiveness of the AI assistant Ivan spoke about the key metrics: response quality, completeness, hallucinations, period errors. Estimates based on manual marking around 600 such requests have been made
- Precision: 0.54
- Recall: 0.43 For LogAnalyzer, the results were as follows.
- Precision: 0.64
- Recall: 0.85 6. Development prospects In the final part of the speech, Ivan outlined plans for the future. Launching new iterations of the SRE assistant with improved capabilities Further improvement of metrics and integration of anomaly detection system Constant search for new ideas and exchange of experience with the professional community
All in all, I liked the report - an interesting balance of general theory and details about the specific scenarios we're trying to AI's for our SRE engineers.
P.S. If the topic seems interesting, you can read more. my review of the article About the general processes around reliability in T-Bank, written by Lesha Merson, ex-dev advocate of the Sage platform.
#AI #SRE #SystemDesign #Software #Architecture #Metrics #SoftwareArchitecture #Engineering