Data - The Land DevOps Forgot • Michael Nygard • YOW! 2023 (Data column)

Interesting. speech from Michael NygardThe author of "Release It!" (I told you about it. her)And now VP at Nubank. (Brazilian neobank)where he is responsible for the direction of data engineering. I wonder what Michael once did. popularize writing approach (architecture decision records). And in this talk, Michael talks about the concept of data mesh, but in an appendix to squawk tasks inside Nubank, where the old ETL/ELT process is no longer scalable. In fact, Michael came there to change the approach towards the concept invented by Zhamak Dehghani, who wrote on this topic. book. The concept itself has four key principles.

1. Domain ownership
2. Data as a product
3. Self service platform
4. Federated governance In fact, this approach can be compared to applying a microservice approach to oltp systems, but transferred to the world of analytics. Vlad Hononov wrote about this in his book “Learning Domain-Driven Design”. I did a brief sammari of these ideas in stand-alone.

In Michael’s speech, the practice of applying this approach is interesting. For example, he explains why it is difficult, for example.

1. Data Ownership imposes obligations on data owners (Cost of data production and support)Benefits are given to consumers.
2. Self-service platform is quite complex to create, but it helps in creating data marketplace (Cost Visibility, Pricing for Data Use)
3. What's missing is the whole story about data mesh: interoperability, "swagger of data," it's all about working with data.

Finally, Michael Raassays on how to approach such changes in data management. (Advice can be viewed in the attached slides).

#Data #Management #DataMesh #Software #SoftwareArchitecture #Architecture #Engineering