Data Mesh: Delivering Data-Driven Value at Scale

Recently, I went to the book club of my colleagues from the data platform to discuss this. books Zhamak Dehghani . . We had to discuss Chapter Nine of The Logical Architecture, which I had read by the time the issue was recorded, skipping the rest of the chapters, partly because I remember the concept from its inception and also saw it parsed in different books.

• In 2019 Zhamak Dehghani has spoken about data mesh at many conferences. QCon Or at the Software Architecture Conference in Berlin, which I personally attended.
• In 2022 year came out book Zhamak Dehghani came The book club goto to discuss this book
• In the book "Learning DDD" Vlad Hononov in one of the chapters dealt with the connection between domain driven design and data mesh (And I have abstract)

In fact, the chapter on logical architecture begins with memories of data mesh concepts, and then there is an attempt to figure out how to implement them.

1. Domain ownership extends domains with analytical data sharing interfaces
2. Data as a product introduces a new architecture quantum, aka data quantum
3. The self-serve data platform drives a multiplane platform architecture
4. Federated computational governance embeds computational policies into each data product

Further, the authors talk a lot about contracts for data from domains, as well as about cross-dependence between domains. In the second, the authors use the ambiguous concept of architectural quantum introduced in the book "Software Architecture: The Hard Parts", which we have published. stripper "Code of Architecture" podcast. The same concept was actively used in the book "Building Evolutionary Architecture", which we also used. stripper

An architectural quantum, defined in Building Evolutionary Architectures (O’Reilly), is the smallest unit of architecture that can be independently deployed, has high func‐ tional cohesion, and includes all the “structural elements required for its function.”

From this concept of architectural quantum, the children are born the concept of data quantum, where ETL is as if inside and abstracted from the external environment. The quantum itself is represented by APIs: input data API, output data API, Discovery and observability API. The following is an interesting discussion of the platform and its individual planes:

• Mesh experience plane
• Data product experience plane
• Infrastructure utility plane Where the most interesting is the first level, as it allows you to share different date products published by the teams separately. Separately, it is noted that the entire platform should be linked policies and standards so that the overall concept does not fall apart due to a decentralized approach.

In general, the approach is interesting, but quite difficult to implement. You can see a report about the data platform of Evgeny Ermakov from toloka.ai, where they in practice went towards data mesh purposefully, although there is still a focus on dbt as the core of their data platform. I did A brief sammari of this report.

#Data #Datamesh #DWH #Processes #Management