Big Data is Dead (Data column)

This one posture Jordan Tigani came out almost two years ago on a blog by MotherDuck, the parent company of DuckDB. Jordan was one of the engineers behind Google BigQuery, and then he was responsible for product development, which means he knows what he’s talking about. MotherDuck is trying to build a cloud-based version of DuckDB, an interesting in-process analytics database described in the report.DuckDB: Crunching Data Anywhere, From Laptops to Servers • Gabor Szarnyas • GOTO 2024 "which I already am." handler. But going back to the article itself, Jordan’s message is that the era of big data as the dominant concept is over. The bottom line is that most organizations don’t actually work with huge amounts of data, and the focus should shift from data size to practical insights. The key ideas of the article are

1) Looking at big data through the lens of reality: The narrative that companies are overburdened with huge amounts of data has fallen short for most organizations. Although the volume of data is growing, Development of hardware and database technologies is outpacing this paceThis makes traditional systems sufficient for many tasks. 2) Most companies have a moderate amount of data: Google BigQuery’s analysis of customer data shows that most organizations manage relatively small datasets, often less than they do. 1 Terabyte. Even large enterprises rarely generate or process truly huge amounts of data, and typical workloads affect only a small fraction of the stored information. 3) Separation of compute and storage: Modern cloud architectures separate storage and computing resources, allowing them to scale independently. This has led to an increase in storage volume with the same computational needs, as analytics is more often focused on current or aggregated data than historical archives. 4) Economic and practical constraints: Processing large amounts of data is expensive and often redundant. Technologies such as column storage, partition cutting and data compression reduce the amount of information processed, which corresponds to economic incentives to minimize costs. 5) Data as a source of risk: Storing large amounts of unused data can pose risks, including compliance problems. (e.g. GDPR)Legal liability and operational difficulties due to the ageing of old data sets. 6) The decline in popularity of big data systems: traditional monolithic databases (For example, MySQL and Postgres) Scalable systems such as NoSQL are stagnating. The number of workloads requiring distributed systems has decreased due to a significant increase in the capabilities of single machines.

Finally, Jordan suggests that companies evaluate their real data needs instead of following the marketing narratives of big data. Most organizations can benefit from smaller-scale data management tools instead of investing in systems for hypothetically huge amounts of information. One of these tools could be DuckDB. (It reads between the lines.).

P.S. In the continuation of the topic can be studied

1. Article "What Goes Around Comes Around... And Around...Michael Stonebreaker, creator of Postgres, and Andrew Pavlo, a database researcher, about the development of DBMS in recent years. 20 years. I have a brief overview in three parts: 1, 2 and 3
2. Speech from the creator of DuckDBA Short Summary of the Last Decades of Data Management • Hannes Mühleisen • GOTO 2024"and my brief sammari
3. PodcastResearch Insights #6 With Kolyov Head about the date of the platform"
4. PodcastCode of leadership #22 Dima Anoshin about data engineering"

#Database #Architecure #Software #Data #SystemDesign #Engineering