Lessons from a Hyperscaler • Casey Rosenthal • GOTO 2024
Interesting. report from Casey RosenthalThe author of the bookChaos Engineering", in which he talks about the lessons learned from working in hyperscale companies, where hyperscale it
Hyperscale is the ability of an architecture to scale appropriately as increased demand is added to the system. He worked for Netflix and was originally part of the traffic team, which was responsible for making Netflix work for users. And in complex systems there is a level of complexity that does not fit in any head. In order to combat this complexity, Netflix invented Chaos Monkey, which randomly disables machines with parts of the system. This helps engineers look for system flaws and make decisions to improve system reliability. Next, the author gives a specific example with two microservices, one of which is stateful and stores data in persistent DB, and he also has a cache in Redis. And then something goes wrong with the stateful service and the whole system deteriorates.) The point of the example is that in complex systems, gaps are possible, even though all components meet specifications.
The author says 4 Models that help hyperscalers deal with complexity:
- Observability (not metrics) Observability of the system, just a metric is not enough
- DevUX (platfoorm engineering) Creating platform teams that make dev2dev products that reduce cognitive complexity (readable my analysis "State of Platform Engineering Report 2023 Puppet")
- Experimentation over testing Experimentation, not testing. The bottom line is that experimentation helps us learn something new about a system, and testing is testing that the system meets our expectations. n
- Low bureaucracy Low levels of bureaucracy, where engineers have enough authority to do their jobs without a bunch of approvals
The author tells us about 4 economic pillar of complexity
- States Limit the number of states of the system. For example, the Ford Model T, where a consumer could buy a car of any color if that color is black:) But now this is a thing of the past and we value the variability and personalization of products for a specific customer.
- Relationships Here, too, everything is growing – new connections are added, new levels of abstraction are added, the dynamic behavior of systems is becoming more complex.
- Environment The environment is also not predictable and few companies can significantly influence it in their favor. (If you are not Amazon or Google)
- Reversability But here the development of software stands out against the general background. We can choose to focus on that and be able to roll back change. Here we help CI/CD, a/b hypothesis testing. But in order for this to work, we need observability.
And the author talks about approaches: observability 1.0 observability 2.0where the observability 1.0 it • Metrics (metric) • Unstructured logs (unstructured logs) • Structure logs (structured logs) Lack of Observability 1.0 is the diminishing utility and increasing costs of creating and storing logs and metrics. The bottom line is that as the system grows, components are added, incidents, logs will become more and more and it will be difficult to find the necessary information in them, and it will take up more and more space.
In response, the author says observability 2.0, which has the following capabilities: It is an approach with high cardinality/high dimenions and in one place that is a single source of truth. From this repository you can get indicators, logs, structured and unstructured data And besides, it does not reduce the effectiveness of developers, but rather increases through the use of one solution.
At Tinkoff, we have a single source of truth. Sage, our obserbility platformThis is the de facto standard for everyone in the company.
#Devops #PlatformEngineering #Management #SoftwareDevelopment #Software #SRE