How Data & Software Eng. Teams Collaborate to Ensure Smooth Data Integrations • Sam Bail • GOTO 2023

Interesting. speech Sam Bail about the collaboration of teams responsible for software development and analytical data:) Slides of this talk are available. here.

The situation looks like this.

Product manager: We’re launching this awesome new feature next month! And we need analytics from day 1! Let’s GOOO! Data team: HOLD ON! Lemme talk to the software engineering team first and see what their data architecture looks like…

And then the report is devoted to the following topics and is built in the form of important questions that are usually asked by Sam in the next project. 1. Logistics We need docks, we need meetings and a clear area of responsibility, clear communications, answers to questions: what we plan to measure and when. (On the first day, week, month...). How to make sure that the development team was in sync with analytics. 2. Infrastructure - where the data is hosted, what type of storage is there, can our ETL tools handle it, whether an SSH tunnel is needed. Are there prod and dev instances, do we use replicas to get data? Do we need access to the tape? What do we do with credentials? (They are personal or common as we shuffle them.). When will we be able to access the data? On dev or prod? 3. Data model What does the data schema look like, is there documentation, who supports the changes in the schema, and who communicates them and how? What will data constraints enforcing look like? (foreign key relationship, NULL values, default values, JSON schemas)? How are time zones handled in dates, currencies? Are we really saving everything we want to measure? 4. Application and data flow How and when are records created and fields filled with values? What actions cause value modifications? How data modification events are logged (updated\ at field or separate table with logging events)? How removals will be handled (hard or soft removal)? Are "old" data being archived? Do I need to migrate data from an old application? Will there be realistic test data to develop? Will there be test data in the production environment? 5. Data contracts - how the agreements from points will be documented 1-4? And how do we enforce them in the future without requiring too much human involvement? What can be taken out of this in the CI/CD and checked on the data producer side (Not as usual on the consumer side.)? How will the changes be communicated and who will be informed? What should I do if something breaks? How will it be necessary to report on the problems, and also what SLA will be on the fixes?

The author summarizes the entire report in three paragraphs.

• Integrating data from a new source into your data warehouse isn’t just “plug n play”
• There are an infinite number of questions to consider. You will probably miss something.
• The key is connection and context between teams.

And then, if you do it right, the problem from the very beginning turns into

Product manager: Look at this awesome new feature! And the dashboard to track all these cool metrics! Data team: Well it’s not everything you asked and it was a bit bumpy getting there, but it works! Go team!

#Data #Software #SoftwareDevelopment #Engineering #Management #Leadership #Databases