The 12 Factor App for Data • James Bowkett • GOTO 2023

Interesting. speech James Bowkett on the topic of good approaches to working with data. In the title he makes a reference to 12 factor, which at one time formulated the guys from Heroku and which became a harbinger of the approaches of cloud native applications. James offers 12 Factors that will help to make the pipeline work with data, which is often called big data and used to train ML models, better:) I liked James’ principles and especially the way he structured them.

-> Architecture & Design Factors related to decision design 1. Data structures as code Universal advice not to rely on UI, but to configure settings to manage data through code. This is standard IaC advice. (infrastructure as a code), GitOps and so on. 2. Append-only data structures The use of such data structures adds historicity and allows time travel. 3. Optimise for access and retrieval The author recommends not to make a data cemetery (data graveyard)Think about how to denormalize data so that it is convenient to use it. 4. Separate data from logic The author cautions against using leaky abstractions (leaky abstraction), like magical meanings that require special processing on the consumer side. This leads to entanglement in the data, as well as a pile of extra noodles in the code from consumers. 5. Strongly type your data columns The author encourages you to think about data types and use them. This allows you to get better data in storage + the storage engines themselves work more efficiently if we do not need to constantly cast data between types. (Which, by the way, is also an abstraction.) -> Quality & Validation Factors related to data quality and validation 6. Architect for regression testability Our solution must be designed to meet the need for regression testing of shipped data, which is a pre-requisite for CDs. (continuous delivery) 7. Track changes in your test data The author recommends storing the change log that was applied to the test data and applying it consistently between environments. -> Audit & Explainability 8. Mind your metadata: Data-Cataloguing The author talks about data cataloging, which allows you to manage metadata. He mentions it in a small way. OpenMetadata and Apache Atlas 9. Mind your metadata: Code Traceability The author recommends tracing from data to code, systems, and the people who generated it. This allows you to understand the origin of data, which can be useful in treadshuting and not only -> Consumption 10. Defined APIs for accessing data The author recommends that you specify the API, separate the internal data model from the external one, and never, ever open access to your internal storage. (Avoid interfering through a database.) 11. Defined SLAs (& SLOs) for data The API must have a defined level of service and expectations for consumers. 12. Treat data as a product Data should be viewed as a product. And then you need to think about the consumers of the product, their needs, use scenarios. As a result, data begins to work and the organization becomes data-driven.

#Data #DataOps #Databases #Software #Engineering #Management #Processes #Devops