Designing A Data-Intensive Future: Expert Talk • Martin Kleppmann & Jesse Anderson • GOTO 2023

It's cool. interview top-level Martin is known by many as the author of a classic book with a boar. ("Designing Data-Intensive Applications")He is now more of a researcher and is engaged in so-called Local-First Collaboration Software. (automerge) You can see more in Martin's speech, which I handler month Jesse is the author of “Data Teams” and the author of “Why Most Data Projects Fail and How to Avoid It.” told 2 month Both participants are excellent in the topic of conversation and are interesting to listen to. It's available here. transcription.

Below I will tell you the main thoughts. Martin talks about his book, DDIA, about the basics of working with data and the wide landscape of technological solutions. (momentarily 10 A few years ago, Martin started writing a book.) Martin plans to write a second edition of the book in which he will address the issues

• The growing popularity of cloud-native applications and the development of distributed applications in layers, when one distributed system relies on the underlying distributed system and so on - I recommend reading more about this in the whitepaper from the guys from Google "Deployment Archetypes for Cloud Applications", about which I told earlier Map-Reduce sunset, whose clearing was taken by cloud DWH in terms of analytics and streaming systems in terms of custom logic applications I like that Martin continues to use the approach in the second book.

So I'm not telling people what to do, I'm telling people what questions to ask. However, writing a new book is not fast, so you should not wait for it soon. Martin and Jesse then discuss the concept of a single tool for all data-related tasks, and conclude that if it did, it would do all of those tasks poorly. So Martin says you need to use the right tools for specific tasks, like starting with a relational database that’s flexible and productive. It is cool to use streaming systems in order to be able to switch to a more convenient solution. For example, you could run both consumers side by side for an amount of time, check the consistency across the two systems, and then eventually decide to switch over from the old one to the new one. Streaming systems allow that so much better than, for example, systems based on, like, doing calls to individual services. So I feel like, as you said, the streaming can help with making change easier there. Martin goes on to say that it’s often important for startups to try to make things as simple as possible, because they’re always in a tough environment of time, people, and resources. And things are changing very quickly and you have to be prepared for these changes, so using Kafka as a streaming platform in a startup can be helpful.

Martin goes on to talk about automerge, which I mentioned at the beginning, and then goes on to talk about working as a scientist and why he switched after that. 10 Years in startups as a scientist and how it allows him to better think through complex ideas in the field of distributed systems. And also why so few PhD dissertations turn into working systems (Significant exception to Apache Flink).

Finally, the participants give advice. I guess my recommendation would be to learn just enough about the internals of the systems that you're using so that you have a reasonably accurate mental model of what's going on there. Jesse: There are a lot of paths out there. Pit's to look out the various paths, look at what you want to do and what your skills are and see if one of those applies to you. Depending on your strengths and preferences, you should choose your path.

#Management #Data #Databases #SystemEngineering #DistributedSystems #Software #Architecture