Skip to content
#SRE

Why Is My App SLOw? Defining Reliability in Platform Engineering • Jez Humble • YOW! 2023

#SRE #SystemDesign #Software #Architecture #Metrics #SoftwareArchitecture #Engineering #Math #ContinuousDelivery

Cool. speech Jez Humble talks about how to determine that the platform has problems, not the consumers who use it. Jez now runs SRE on Google’s serverless platform, which has a large number of customers with different load profiles, different latency metrics, and so on. In this configuration, there is a problem in separating client load problems from platform problems. But the guys at Google did it using their mathematical stats. To be more precise, the goal was

A metric that represents the customer experience

  • Combinable across projects / cells / regions
  • Can be used to detect anomalies affecting multiple customers (likely platform issues)
  • Computationally cheap (high QPS)
  • Principle-based The author discusses the three pillars of reliability:
  • Availability The basis here is to calculate the number (share) unsuccessful requests. The situation is complicated by the fact that there is some subjectivity in mistakes. (a lot of 400x and 500x errors that are set by the owners of user loads)There may be errors due to broken deadlines, there may be errors in poorly formed requests from customers, retries may increase the number of errors.
  • Performance - here the author talks about installing P99 latency SLO and creating probes. But there are problems with dependencies between loads and the fact that the testers can be too narrow.
  • Correctness There are many tests and analysis of canary deployments, but the problem is that coverage is limited.

Next, the author says that you can try to build a distribution function for user loads. (For example, log-normal)And then we can imagine that user loads can be considered stationary. And then use the two-sigma method and formulate a hypothesis.

Hypothesis: Self-Similar Workloads Should Have Consistent Performance Technique Overview:

  • Partition workloads into Cohorts ← Approximate Intent via Workload Features
  • Build Performance Baselines ← Estimate Distributional Form (e.g. Normal)
  • Estimate Likelihood of Delivered Performance ← Test For Stationary Result:
  • Set of Events with Predicted Likelihoods
  • Time-series of summary statistics describing concentration of extreme outliers In addition, the strategy of using
  • Calculate z-scores among the loads by kahorts, where z-scores = (observed-workloads - baseline-mean) / baseline-std
  • Track the proportion of loads with z-score ≥ 2 time-window
  • Consider that 2-5% percent of loads with 2σ deviations of the norm
  • Trigger when this indicator is higher 10%

There are many more interesting points about how this works and why, but the important conclusion is that this approach allows for reliable detection and measurement of the impact on user loads of problems on the platform itself.

P.S. I like Jez's materials, which come in both print and report form. I told you before.

#SRE #SystemDesign #Software #Architecture #Metrics #SoftwareArchitecture #Engineering #Math #ContinuousDelivery