Root Cause: Stories and Lessons from Two Decades of Backend Engineering Bugs (Books column)
I read a few chapters. books Hussein Nasser, a popular blogger (500k subscribers channel)What does it say about software engineering? I took it out of mere interest to read not just about architecture, databases, and distributed systems, but rather how it all breaks down in the marketplace. In fact, the author promises these stories in the description of his book. The author believes that the experience of an engineer is measured not only by years, the number of features or lines of code, but also by the number of bugs that he was able to reproduce, track down to root cause and fix. It sounds almost banal, until you remember how much real engineering knowledge comes from such investigations: somewhere I learned how the browser restricts HTTP.1.1 connections, somewhere figured out HPACK in HTTP/2It was the first time I saw a file descriptor limit. 504 Gateway Timeout.
This book reminded me of the experience when I myself worked with incidents, wrote or disassembled post mortems, and also managed the treadshooting sections for SRE-engineers who wanted to get into the company and eventually made a couple of public appearances and articles. (They are available on my website system-design.space: Description of the interview approach and sample).
If I go back to the book, I have read a few chapters, and this is what they are about.
1A story about system slowdown. Users can see that the whole product is slow. Not one endpoint, not one button, not one script, but as if everything is bad. In the end, the investigation leads to a small UI element: the search bar shows text like “Search 550M items”, and JavaScript pulls the API that performs SELECT COUNT.(*) on a huge chart. He does it over and over again. There's more in the Trace. 100 Thousands of such requests 30 minutes.
The good thing about this is that a simple answer to add a CPU base would be wrong. Yes, the base was overloaded by the CPU. But the reason was not that the base was “weak,” but that the grocery UI generated a meaningless backend load. The real fix is not to scale the symptom vertically, but to remove the very need to count the exact number where the user is approximate enough.
2Stories about live translation system, HTTP/1.1, HTTP/2 load balancer And that seems like an interesting part for architects and tech executives. At first, long-lived SSE sessions rest on the browser limit of connections per host: the seventh request simply hangs in the client-side queue. Then turn on HTTP/2 It solves this problem, but brings another: a lot of small queries, TLS, frame parsing, stream state and HPACK start eating the CPU backend. Then the load balancer appears, the backend is translated back to HTTP.1.1Static resources are cached, things get better – until the unlimited backend connection pool comes up against file descriptors and TCP receive window.
That is, the book shows the unpleasant but real picture: architectural improvement is rarely “just an improvement.” It changes the bounce profile. HTTP/2 Remove one bottleneck, but add CPU overhead. Load balancer unloads the backend, but brings new settings and new default values. Increasing the file descriptors limit helps, but does not explain, why so many connections were created at all.
In general, the book is really good at describing the reality of treadshooting problems all over the stack and finding root cause problems. It will be useful for engineers who want to not only talk about cool architecture, but also understand how different components can work with each other if something goes wrong. As a result, the book is more like an engineering incident blog than a regular academic tutorial, and can be useful for engineers to prepare for the same interviews.
#Books #Engineering #Software #Architecture #Management #SRE