Skip to content
#SRE

Technoshow "Dropputo": release 1 (Category SRE)

#SRE #SystemDesign #Software #Architecture #Metrics #SoftwareArchitecture #Engineering #Databases #Data

I saw it yesterday. show-off A new show from my colleagues at T-Bank. In this show, the guys talk about incidents on data platforms, with an honest analysis of the reasons and conclusions. Interestingly, the show is broadcast live, which should inspire hope in the minimum of editing and f... The format is in the spirit of blameless postmortems, where we approach postmortems without charge and even with a focus on learning - to talk about why failures happen, how to prevent them and what to change in the process/architecture to improve reliability (By the way, the reliability of T-Bank was well told by Lesha Merson in an article about which I wrote earlier).

The first issue of the podcast featured Alexander Krasheninnikov from T-Bank, a data platform/ETL/DWH practitioner, as well as a man with a good sense of humor and improvisation skills, as can be seen from his account of the two-week data platform incident he recounts in this episode. The chain of incident looks like this: deletion / loss of metadata → reading drop in Trino → no backup / slow CDC recovery → critical incident → reversal to the new Kafka architecture + contracts → unification of schemes, parallel downloads → validation and correction of discrepancies → recovery of service with possible partial loss → restoration of service from backups and reloading of historical data.

Engineers may like this "combat" analysis of the incident

  • Operation of CDC (change data capture): where Debezium is convenient, its architectural variants, typical rakes Integrity Patterns: An outbox approach for secure event exchange between services and limits of applicability. Pipeline Observability: Which SLIs to Raise for “Data Speed” (freshness/latency)integrity (duplicate), connector stability.

The episode is useful for management optics: Common language of risk: linking the color/seriousness of an incident to error budgets and release frieze Disruption Learning Culture: blameless post-mortems as a system tool for quality and communication between data/product/SRE teams.

  • SLO management: translation of user metrics (For example, “the data in the window is fresh ≤ X minutes 99,9% of the time) (c) the risk plan/fact.

#SRE #SystemDesign #Software #Architecture #Metrics #SoftwareArchitecture #Engineering #Databases #Data