dbt - the core of a modern data platform - Evgeny Yermakov - SmartData 2023 (Category Architecture)
Interesting. report Yevgenia Ermakova about the construction of a date platform in toloka.ai, which, having gained independence from Yandex, was forced to move to new technologies. In the end, the choice fell on databricks, dbt, airflow and tableau. The author explains why this choice was made and how it all works.
The highlights are as follows: Toloka itself is a crowdsourcing system, where customers come with tasks like marking up data, and on the other hand, people who perform them are registered on the platform. Architecture is based on three pillars: -- Data lakehouse Processes in accordance with the data mesh approach
- Modern technological stack Before moving to new technologies, the guys used a lot of their own, some of which are already in opensource: YTsaurus, datalens After the move, new technologies were chosen and dbt became the core of the system, closing the functionality: data quality, data catalog/data observability, batch processing (spark), orchestration (airflow)
- Originally dbt. (data building tool) It was needed as a convenient tool for the transformation step in E.TL/ELT It is interesting that in the concept of dbt there is an opinion on the roles, where in addition to the standard data engineers and data analysts, there is also an analytics engineer. As a result, data engineers are those who make the data platform work efficiently, data analysts look for insights in data and help them use it effectively, but analytics engineers are guys, something in between the other two + fits well into the concept of data mesh, where there is no centralized data command, but data commands by domains. The basis of the dbt project is the dbt model. The model consists of a file describing logic. (.sql or .py file) and a file describing the configuration. In the .sql file there is a request to form an object, other models are used through ref() source() + used jinja template. In the .py file return the dataframe with the calculated data, there is access to all the features of pyspark + other models are also used through ref() source() The materialization of the query dbt takes over and there are different strategies, of which the most interesting incremental. Settings are stored in dbt\ project.yaml and profiles.yaml
- dbt supportive A large number of databases, such as postgres, mysql, clickhouse, etc. dbt is a console utility, for example, when starting dbt build, all dependencies between models are assembled, as well as compiling python / sql queries and writing to manifest. json The dbt run command runs compiled queries, where you can configure the start in different ways, but it is interesting to run by graph By the way, dbt can generate documentation with the command dbt docs generate and then you can see the lineage data. We can also write tests in the same place where we describe models, and then run them using dbt tests. For example, we can check unique or not null on the field, as well as if we want relations between models. DBT has many more options, but you should read about them yourself:) Next, the author explains how to make data mesh at the dbt + airflow level. The author considers variants of the type: Monolithic - one dbt project for the whole company Microservices - individual dbt projects for each domain Layered - individual dbt projects by level Mixed – Anarchy, where projects are created by whoever wants We chose a monolithic approach and got ala monorepo under data mesh, in which everyone lives. This was due to the fact that the microservice approach broke all the links between the models. (before 1.6 There was a problem with importing each other, as this led to cyclical dependencies.). Another interesting thing was the converter of the dbt execution graph into airflow format to run DAG from airflow.
As a result, the guys implemented their approach to data mesh with the help of open source tools and the whole scheme looks quite slim.
#Data #Datamesh #DWH #Processes #Management