[1/3] What Goes Around Comes Around... And Around... (Data column)
Interesting review article 2024 yearly Michael Stonebraker and Andrew Pavlo Database development in recent years 20 years. Both authors are korepheus in the field of databases: Michael Postgres Andrew is a database researcher, professor and lecturer whose lectures accessible Youtube.
The article continues the article 2005 yearWhat Goes Around Comes AroundIt was written by Michael Stonebraker and Joseph M. Hellerstein. They analyzed the history of database development for 35 They predicted that then fashionable object and xml databases would not be able to bypass the relational model.
It's been an order since then. 20 It’s time to take a fresh look at the world of databases. To do this, the authors decided to look at it from two sides:
- Data models and query languages
- Database architecture
Let’s start with the analysis of existing data models and query languages:
1. MapReduce systems In the beginning. developed Google for processing large amounts of data (webcrowler). MapReduce does not use a fixed data model or query language, they perform custom map and reduce operations. The open version of MapReduce was Hadoop, which is now not very popular due to low performance and is replaced by more modern ala platforms. Apache Spark Or just a DBMS.
2. Key-Value Storage They have the simplest data model: key-value pairs. They are used for caching tasks. (Memcached, Redis) or storing sessions. Opportunities in the model are limited - there are no indexes or join operations, which complicates application for complex applications. Many KV storage facilities (for example, DynamoDB, Aerospike) Evolved into more functional systems with partial structure support (JSON). Among the popular built-in k/v solutions are popular LevelDB and RocksDB.
3. Documentary databases They store data as documents. (For example, in JSON format.). Originally popular due to the ease of integration with web applications (for example, MongoDB)It offers a schema on read approach. Interesting what to 2020Most document databases have added SQL-like interfaces and support for ACID transactions, and sometimes schema on write.
4. Column-Family databases (wide columns) In fact, this is a simplified version of the document model with limited nesting. Beginning. Google BigTable and the world has an open source implementation. Apache Cassandra. Initially, Cassandra had no secondary indices or transactions. But as they developed, they appeared. (It's all there. very interesting)
5. Search engines They are needed for full text search. (Elasticsearch, Apache Solr). Support text indexing, but are limited in transactional capabilities. Relational databases also offer built-in full-text search, but with a less user-friendly API.
6. Array databases They are designed to work with multidimensional arrays, for example, scientific data. (SciDB, Rasdaman). The niche is limited to specific areas of application: geodata, genome study.
7. Vector databases Used to store embeddings from machine learning (Pinecone, Milvus). The main application is to search for nearest neighbors in high-dimensional spaces. Relational databases have already begun to add support for vector indices.
8. Graphic databases Model data like graphs (nodes). Examples: Neo4j For OLTP graphs, TigerGraph for analytics. Most graph problems can be implemented on relational databases using SQL/PGQ. (standard SQL:2023).
General conclusions Most non-relational systems either occupy niche markets or are gradually converging with relational databases. SQL remains the main query language due to its flexibility and support for modern applications. Relational DBMS continue to evolve and integrate new capabilities (For example, JSON, vector indexes)This makes specialized systems less competitive.
In continuation This post is about database architecture.
#Data #Architecture #Software #DistributedSystems