Apache Iceberg: What It Is and Why Everyone's Talking About It (Data column)

Excellent. brief Tim Berglund's VP Developer Relations in Confluent is about Apache Iceberg and why it's been talked about so much. To put it simply, Apache Iceberg is an open analytical table format that brought ACID transactions and the evolution of the scheme to the world of raw data lake. In this video, Tim, in addition to Iceberg, also talks about TableFlow from Confluent, which is an automatic materialization of Kafka-topics in the Iceberg table. Here are Tim's main points

1. He starts with story about the evolution of approaches datawork

• Data warehouse (DWH) : 1990-x, night ETL, strict schedule, next day reports
• Data lake: c 2010-x, first Hadoop, then S3/Blob; circuit on read, ELT pattern, scale to petabytes In the transition to data lakes lost: strict scheme, transaction and consistency in parallel records

2. Problems in data lakes were about Lack of metadata at the file level - this led to long directory-list operations in S3; unpredictable SQL queries No ACID transactions - this led to "dirty" data in partial overwriting, race conditions

• Complex evolution of the scheme - obtained "zombie" columns, as well as complex rolling of parts by hand

3. But I came to the rescue. Approach to Apache Iceberg Which brought layers that solved the previous problems. These layers look like as the level of abstraction increases - Data Files (Parquet/Avro/ORC) - immutable pieces of data. - Manifest Files JSON files listing specific Data Files plus column statistics (min/max, null-count). - Manifest List A set of manifests for a single file addition/deletion operation - Snapshots Atomic states of the table; each snapshot points to a specific Manifest List, thereby forming a point in time. - Metadata File (metadata.json) - keeps a list of all snapshots, scheme, class orders. - Catalog external service (Hive Metastore, JDBC-DB, REST catalog) Compare the table name with the current Metadata File
4. Such a scheme provides the following properties and mechanisms that solve the above problems: - ACID transactions - is provided by copy-on-write + optimistic concurrency, and gives it a safe parallel INSERT / DELETE - Time Travel - is provided by reading snapshot-ID or timestamp, and gives audit and reproducible query

• Schema evolution - is provided by full-text JSON metadata; column identifiers, not their order, and makes it possible to add / rename columns without overwriting - Hidden Partitioning This is achieved through computable transformations. (bucket, truncate, day)What is hidden from users, and gives fast scans without manual filters - Row-level Deletes Provides V2 spec: delete deltas with positional links, and gives GDPR-delete and upsert-patterns

5. Speaking of physicalitythen Iceberg is not a server in itself. This is a specification + libraries (Java, Python, Spark, Flink, Trino, Hive) Metadata and data - ordinary files in the storidge object; directory - "plugin" (Hive Metastore, AWS Glue, REST). Failure-tolerant scheme without rename/list operations is important for S3, GCS.

I’m not going to talk about Tim’s TableFlow part, because the most important part for me was the Apache Iceberg part. If we take stock, Iceberg solves three historical pains of data lakes - ACID transactions, managed scheme evolution, consistent snapshots. The architecture is based on simple JSON/Parquet files and an external directory; no Iceberg server is required. The ecosystem is growing: Netflix switched to Iceberg-only lake (≈1 EB)Databricks acquires Tabular to converge Delta️️Iceberg Iceberg is becoming the de facto standard for open tabular formats, and in the coming years data platforms will converge to a unified lakehouse stack, where Iceberg plays the role of a “common language” between streaming systems and battle analytics.

#Data #Dateabases #PlatformEngineering #Software #Architecture #Engineering #DistributedSystems