A Short Summary of the Last Decades of Data Management • Hannes Mühleisen • GOTO 2024 (Data column)
Interesting. speech from Hannes Mühleisen The history of data management systems and where they will develop. Hannes himself is a CS scientist and creator. DuckDB And the DuckDB Lab cofounder.
Here are the highlights of the speech The author begins with references to a deep history, when clay tablets were used for accounting in the household and this was before abstract writing appeared. Next, the author immediately moves on to the role of IBM, whose history began with the US census. 19 century and beginning 20 A century after the merger of several companies IBM:) And the rise of IBM as a computer giant began with mainframes, which were used for astronautics in the United States. IBM came up with a hierarchical system for managing data and it was used in these mainframes.
- Next. Edgar Frank "Ted" Codd came up with the basics of a relational data model, in a scientific articleA Relational Model of Data for Large Shared Data Banks"
- In 1973 IBM made its IBM System R, which was a pilot implementation of the relational model. But in 1979 Oracle IBM on a turnaround with Oracle's industrial relational base
- In 1983 IBM also entered the race with its IBM DB.2
- And now 1996 In the course of the academic project was created PostgreSQL. He made it. Michael Stonebraker Berkley For those interested in the genealogy of different DBMS may be interesting Map of RDBMS Genealogy
Somewhere in 80It became clear that one base does not cover all scenarios and there was a fork in the form
- Transactional In this scenario, users work with transactions. The data here is efficiently stored line by line (row-based)Because you usually need a lot of writing to read and update.
- Analytical In this scenario, users analyze data, there is no need for transactions. Data here is efficiently stored column (column-based)Since values from specific columns are often needed to calculate indicators, when storing them on columns, they are easier to compress during storage, and also read for calculations. In order to show the differences between the DBMS, the author gives a metaphor for a pickup truck and a machine formula. 1. A pickup truck is a reliable workhorse that should always work, and it is the transaction database. Formula machines are analytical bases that need to be fast, but may not always work, as they usually do not block the entire operation of the company.
Next, the author talks about the No SQL movement!
- 2006 year Map/Reduce From Google to parallel processing tasks on unreliable hardware, Hadoop grew out of this approach. Originally there was no SQL, but 2010 Hive, which returned SQL over map/reduce, because without SQL it was too difficult to write queries.
- 2009 year Schemaless MongoDB. Here the concept was that it was not schema on write, which was controlled by the database, but schema on read, which was supposed to check the app developer. In 2017 MongoDB introduced schema validation
- 2008 year no ACID Transactions and instead tunable consistency in Cassandra. In 2023 The year Cassandra transactions appeared
- 2014 There is no internal storage in Apache Spark. In 2024 In addition, internal storage
The change was due to the fact that the trio
- Tables.
- SQL
- ACID It's too convenient to develop applications, and that's where data will go. The author recalls Postgres and SQLite, as well as newSQL as an approach that combines this trio with scaling as in NoSQL. And then he says that relational databases will eat up almost all scenarios and shows how to do it with key/value, document, time series, vectors, graphs, data frames. But, for example, full text search does not fit well into the relational model.
Finally, the author says that big data is dead, because now we have the opportunity to build a very powerful machine on which to start processing what used to spin in a distributed system. And it might be much more effective.
#Data #Architecture #Management #History