Reliability and Security: Foundation or Optional?

Alexander frames the problem with numbers. With no statistics of your own, external ones will do: an IBM study putting the average cost of a breach at about five million dollars, and a survey in which an hour of downtime costs more than three hundred thousand dollars for 90% of the companies polled — not cascading disaster, just an average. Then come reputation and customer trust, one of a financial organisation's most valuable assets, and regulators, to whom you must account for major outages and leaks. Why now? Systems have grown complex: a multi-cloud of own data centres and cloud workloads, the microservices boom a decade ago that shifted complexity onto operations, and the internet of things, with its enormous attack surface. The slide shows a dependency graph from the Jaeger tracing system on Uber's engineering blog; Netflix, he says, has a similarly insane one from around 2016. Protecting such a system after the fact is next to impossible.

His footing is a Google research paper on software quality. In its model a product has internal properties — how development is arranged and which quality attributes the system ended up with: how easy it is to maintain, evolve and test. And external ones: how convenient the product is, what it does to the brand and sales, and beside them reliability, whose failure a downdetector and the press report immediately, and security, whose problems sometimes keep large companies offline for weeks. Everyone wants the external properties, yet they come out of the left-hand cube. As the curator of architecture he is regularly asked to measure everybody with architecture metrics — but metrics are easy to game, and the paper's authors avoided them too, choosing the process level: a factory under ISO 9000 gets good sausage out of a correct conveyor. That is emergence: system-level properties are not assembled from component properties and cannot be bolted on afterwards. Beside it he shows a fractal: a simple function with remarkable properties.

By reliability the industry usually means SRE practices: designing systems on the assumption that failures happen — redundancy, degradation, fault tolerance. Google has popularised that approach for some twenty years, and T-Bank has its own caste of SRE engineers, though teams work on reliability too. Security went a similar way: prefixes were added to DevOps — DevSecOps, DataOps, MLOps — and those names are what Alexander dislikes, because the substance is lost behind a pretty word. He then walks through a Google whitepaper on security by design. The logic is simple: code not yet in the main branch, code merged, code in production — and security has to move left, to where the system is still being designed and written. That is done with a safe development ecosystem: the practices are built into languages, libraries and components, and the engineer is handed a safe spade instead of a demand to know everything about security. The second cube is safe deployment and runtime control at the control plane, where security people like to invoke Zero Trust; the third removes standard vulnerabilities from templated applications.

Then Alexander points at the hole in design work. For a while he owned how the company runs its system design interview: the candidate gets a simplified system, business requirements and an incomplete set of quality attributes, and is watched as he gathers the rest, balances them and works out what matters more. It reminds him of the architecture analysis method invented twenty years ago or more at the software engineering institute at Carnegie — ATAM: business drivers, scenarios, quality attributes, comparison and deliberate trade-offs, to which risks are worth adding. But neither that interview nor the company's internal RFCs and architecture decision records really cover security: a decision is worked out at the level of what is being built and which non-functional requirements it must meet, and then the security people arrive and point out what nobody thought about. Engineers acting as architects, he argues, should also describe what must not be in the system — the threat models. Otherwise the bottleneck is overloaded security engineers reverse-engineering someone else's architecture during an audit.

Among the big companies he starts with Google: reliability at scale, the popularisation of SLAs, SLOs and SLIs, and a separate team of developers made to automate operations, which not everyone can afford. For this talk he made a post with links, among them his earlier write-up of how Google improved its architecture design review process. Netflix went another way: it has a great many services whose reliability is hard to assess, so it took up chaos engineering and fault injection inside its own systems, starting with Chaos Monkey. The approach caught on in the Russian market too — he recalls the Yandex Taxi team describing it at an architecture conference whose programme committee he sits on. At Netflix the engineering teams own what they wrote and how it behaves in operations: when you are constantly tested in production, it is silly not to design for virtual machines, the network or a whole data centre going away. From AWS he quotes the variously interpreted thesis that security is everyone's job, plus the reports for top management that T-Bank has as well.

Since 2020 T-Bank has been moving from scattered infrastructure teams towards an internal platform called Spirit. Its principles: repeatability and transparency of the code lifecycle, quality checks at every stage, repositories open by default since closing one takes effort, and a focus on lowering cognitive load: knowing requirements, distributed systems, security and infrastructure equally deeply is unrealistic, so the right path has to be the easy one. Reliability and security rest on a shared observability platform of logs, metrics and traces, which the company started before import substitution became fashionable and now sells; NSPK uses it. On top sits incident management: services and teams with their dependencies, detection against SLOs, an incident lifecycle, escalation from the team to the technical director and the whole company, blameless analysis, reports, and correlation of releases with failures. Satisfaction with the platform was measured first through Developer Experience and then through SPACE, and one insight is that SREs are the least happy, because they meet production problems most often.