[1/2] Monarch: Google's Planet-Scale In-Memory Time Series Database (Category Architecture)

I finally finished reading the original. whitepaper About this interesting system for working with time-series data from Google. The idea came from my immersion in architecture. whitepaperThe system was redesigned using a quality-based approach and UML diagrams. In a nutshell, the point is that in 2020 year article on VLDB Endowment About this world's largest time series base

1. Monarch is a globally distributed, multi-tenant in-memory time-series database for monitoring almost all of Google’s user and infrastructure services. It receives terabytes of metrics every second and processes millions of queries. Architecture is divided into regions with global plane queries and configurations.
2. Key to scale: lexicographic sharding by “target”, aggregation at the collection stage, compact indices of tips (Field Hints Index)Two-tiered requesters (Root/Zone Mixers).
3. For July. 2019almost 950 billion rows of RAM (~petabyte of compressed data)average collection aggregation 36:1 (before 1 000 000:1)Indices allow you to cut fan-out to tens of thousands of extra leaf nodes.

How this roughly worked The system is multi-tenant and global. Regional zones autonomously receive and store data in memory, and global planes provide a single request and configuration. This reduces dependence on external persistence and increases the availability of monitoring during incidents. Data and query models differ from Borgman’s predecessor (By the way, it was Borgman that became the prototype of Prometheus - you can see this in the documentary I'm talking about. told). Unlike the line keys of previous systems, Monarch uses a type-saturated relational model of metrics. (including distributions/histograms) Expressive query language, which simplifies static analysis and optimization.

• The processing architecture looked something like this. Inguest: customers → ingestion routers → area → leaf router → Leaves (in‑memory). Collection aggregation can already work here. Requests: Root Mixer distributes to zones (Zone Mixers); Index Servers (Including the Field Hints Index) pre-exclude irrelevant nodes, sharply reducing fan-out.
• Separately, optimizations were made. Collection aggregation: average 36 entrance rows 1in the extreme to 106:1. Saves memory/CPU and traffic. -- Field Hints Index (FHI): zones with >10 000 leaves and trillions of keys; FHI allows you to cut off73 000 irrelevant leaves for complex samples. Lexicographic sharding by target: all metrics of one object fall on one leaf → local aggregations / joins, less fanout.

Continuing storyWhat happened to this system next?

#Software #Architecture #DistributedSystems #SRE #Engineering #Databases #Data