back to the presentation
concise summary2019

Postmortems or how we learn from facaps

As the product grows, a failure is no longer the fault of one person or one team. It contains code, shared services, infrastructure, the release process, and incomplete knowledge of dependencies. A postmortem is needed to turn this complex episode into a general model of the system and concrete changes - without looking for a culprit and without a ritual document for the sake of a document.

September 27, 2019Conference6 min read

The summary is compiled from the published Tell Me About Tech transcript, slides and recording. The material has been condensed and edited—it is not a verbatim transcript.

The main thread
01

Blameless - a condition for reliable parsing

If the purpose of the meeting is to find and punish someone who has made a mistake, the participants become defensive, hide their doubts, and simplify the story. The organization loses data on why the action seemed reasonable at the time. The Blameless approach comes from a different place: people acted within the available information, tools and pressure. It is necessary to understand what conditions made the error possible and why the system did not stop it earlier. Responsibility remains, but is transferred from the individual to improving the circuit.

A good postmortem doesn’t measure severity with fancy text or stop at the phrase “human factors.” It captures the impact on users and business, participants and affected components, sequence of events, detection, response and recovery. Then separates the immediate trigger from the root and associated causes. This chronology creates a common understanding between development, operations, and owners of neighboring systems.

02

From incident to closed action items

The process begins with criteria: which incidents require separate analysis and who initiates it. The owner collects the facts and timeline, the participants jointly test the assumptions, and then the document is published to the affected teams. Facilitation is important: the discussion should return to the design of the system, and not to the assessment of people. The template lowers the threshold and helps you remember impacts, causes, successes, and discovery gaps.

The main solution is a small set of action items with owners and deadlines. They can prevent recurrence, detect a similar failure more quickly, limit the blast radius, or improve recovery and communication. The abstract “be more careful” is not an action. The item must change code, configuration, monitoring, test, documentation or process.

03

What real cases showed

Analysis of the unsuccessful release showed that fixing the defect alone is not enough: we need a more secure delivery, a clear rollback and monitoring of key signals after the release. In another case, the dependence on a shared service turned out to be more critical than the teams thought. The incident made visible not only the technical connection, but also the absence of the owner, the agreements and the degradation scenario. The dependency map has become as important as fixing a specific component.

The infrastructure case with network configuration reminded us that infrequent manual operations are especially dangerous: knowledge lives with a few people, verification is incomplete, and an error quickly affects many products. Solutions are shifting towards automation, peer review, auditable changes and incremental implementation. The common lesson of all stories is that the value of an accident is not for its unique cause, but for the opportunity to strengthen several protective layers and spread knowledge beyond the affected team.

Takeaways

What to take away

  1. 01Blameless culture is needed for truthful data about the system, and not for the abolition of responsibility.
  2. 02The postmortem should contain the impact, chronology, discovery, response, causes and contributing factors of the failure.
  3. 03An action item has an owner, a duration, and an observable change; calling for more attention does not prevent recurrence.
  4. 04Training is completed only after tasks are closed and findings are distributed among dependent teams.
Share