[2/3] What Goes Around Comes Around... And Around... (Data column)
Continuing story article 2024 It is worth talking about the different approaches to system architecture that have developed over the past two decades. These architectures reflect changes in software. (Application types and requirements have changed)changes in hardware, or rather changes in hardware parameters (CPU power, RAM speed, different types of disks, network speed and so on).
1. Column systems (columnar systems) These systems have become the de facto standard in the analytical database world. The bottom line is that queries often address only a small number of table attributes. Column storage allows you to read only the necessary data, as well as better compress them due to the uniformity of values. In addition, it is possible to achieve faster execution of requests through vectored data processing. Examples of such databases are Vertica, Amazon Redshift, Google BigQuery, Clickhouse, DuckDB.
2. Cloud databases (cloud databases) These databases are provided in the cloud, where it was fashionable to move your infrastructure. Such databases provided scalability and elasticity, which allowed customers to dynamically change resources. The rise of these databases is due to the fact that 20 Over the years, the network speed has increased much faster than the disk speed, so the use of NAS (network attached storage) It has become an attractive alternative for a standard storage device. Actually, the main vendors in the quality of NAS use object storage (For example, AWS S3). This architecture leads to the fact that we separate compute and storage and we have such advantages. It is possible to provide elasticity at the level of individual requests Computing nodes can be sent to other tasks if the DBMS is not fully disposed of. You can move the computation right to the nodes of storage — the approach is called “pushing the query to the data” and it performs better than the standard “pulling the data to the query.” Interestingly, the first two approaches are called serverless computing and brought them to the world of DBMS. Snowflake. Among the most popular cloud databases Google Spanner – I highly recommend it whitepaper Amazon Aurora – I highly recommend it. whitepaperwhich we've even somehow stripper In the edition of the podcast "Code of Leadership", and I recommend you to look at it. story From AWS Re:Invent 2023 about how they brought the sharding there
3. Data lakes (Data Lakes) and Lakehouses Cloud platforms have ignited interest in abandoning monoliths in analytics and moving to data lakes, where data is uploaded to object storage. (S3 like). That is, there was a departure from the standard earlier ETL. (extract-transform-load) ELT (extract-load-transform). After downloading, calculations were performed directly on top of the data using lakehouse engines, bypassing the standard DBMS. (type Greenplum). Lakehouse is a combination of data warehouse and data lake.) Data in the data lake is stored in binary form: Parquet, ORC (optimized row columnar)For in-memory data exchange, you can use the format Apache Arrow. Each cloud has managed data lake services, and there are separate options in the form of Databricks, Dremio, PrestoDB, Trino.
4. NewSQL systems In the 2010s, this class of systems appeared to combine ACID transactions from RDBMS and the scalability of NoSQL solutions. One of the first representatives was the already mentioned Google Spanner. In fact, it was here. 2 In-memory and standard disk-oriented systems. Some startups bet on a greater demand for in-memory approaches, but the bet did not play. These approaches were replaced by distributed and transactional SQL RDBMSs. TiDB, CockroachDB And I think we can add it. YDB with their Calvin transactions. These systems are suitable for applications that require both high performance and strict data consistency.
In continuation ending of the article.
#Architecture #Software #DistributedSystems