The table format you pick decides how transactional your data lake is, which engines can query it, and how hard it is to leave your vendor. In 2026 the realistic shortlist is Apache Iceberg, Delta Lake, and the newer DuckLake, three formats with genuinely different metadata designs and lock-in profiles.
What is an open table format, and why does the choice matter in 2026?
An open table format is a specification layered over Parquet files in object storage that adds ACID transactions, schema evolution, time travel and safe concurrent writers. The choice matters because it determines which query engines, catalogs and vendors can read and write your data. It is the deepest lock-in decision in a lakehouse.
Plain Parquet directories fail quietly: no atomic commits, no consistent reads during writes, expensive file listings, and schema drift nobody notices until a dashboard breaks. Table formats fix this with versioned metadata, which is why a single copy of data on S3 or MinIO can now serve Spark, Trino, Snowflake, ClickHouse and DuckDB simultaneously. Apache Hudi still exists and is maintained, but most new evaluations in 2026 have narrowed to the three formats compared here. The stakes rise as AI workloads push more critical data into lakehouses; the retrieval-quality arguments in RAG and Data Engineering in 2026: Your AI Is Only as Good as Your Data all assume a consistent, queryable lake underneath.
How does Apache Iceberg structure its metadata?
Iceberg stores metadata as files beside the data: a root metadata JSON per table, manifest lists per snapshot, and manifests listing data files with column statistics. A catalog holds a single pointer per table and swaps it atomically on commit. Any engine speaking the Iceberg REST catalog API can plan queries against that tree.
Strengths and costs of this design:
- ▸Engine-neutral by construction. The spec is Apache-governed, and the REST catalog API has become the closest thing the lakehouse world has to a standard interface.
- ▸Rich statistics enable pruning. Manifests carry per-column min and max values, so engines skip files without opening them.
- ▸Format v3, finalized in 2025, added deletion vectors, row lineage and a variant type for semi-structured data. Check per-engine support before relying on v3 features.
- ▸Maintenance is your job. Snapshot expiry, orphan-file cleanup and manifest compaction must run somewhere; neglected Iceberg tables accumulate thousands of small metadata files and planning slows down.
- ▸High-frequency small commits contend. Optimistic concurrency retries hurt when many writers commit to one table every few seconds.
How is Delta Lake different under the hood?
Delta Lake keeps an ordered transaction log, JSON commit files plus Parquet checkpoints, inside the table directory itself. Readers reconstruct table state by replaying the log. No external catalog is required for correctness, but concurrent writers need an atomic commit primitive on the object store, historically the fiddly part on S3.
What that means in practice:
- ▸Gravity sits with Databricks and Microsoft. Delta is the native format of Databricks and of Microsoft Fabric's OneLake; inside those platforms it is the frictionless choice.
- ▸The log-in-directory model is simple and robust for single-writer and Spark-centric pipelines, and checkpoints keep log replay fast.
- ▸UniForm narrows the interop gap. Delta tables can emit Iceberg-compatible metadata alongside the delta log, letting Iceberg readers consume Delta tables, read-only and with feature caveats worth verifying.
- ▸Non-JVM writers exist but trail. The delta-rs implementation for Rust and Python covers common paths; parity with the Spark writer is not guaranteed for newer features such as liquid clustering.
- ▸Governance is Linux Foundation, direction is one vendor. The project is open source, but the roadmap overwhelmingly follows Databricks priorities.
What does DuckLake do differently with a SQL catalog?
DuckLake, published by the DuckDB team in 2025, moves all table metadata out of object-store files and into an ordinary SQL database such as SQLite, PostgreSQL, MySQL or DuckDB, while data remains Parquet on object storage. Commits are plain SQL transactions, which makes multi-table ACID and low-latency planning straightforward.
The consequences are bigger than they sound:
- ▸Cross-table transactions. Iceberg and Delta commit per table; DuckLake can commit a change spanning many tables atomically because the catalog is one database.
- ▸No metadata file explosion. Thousands of manifest files become rows in indexed tables, and query planning is a SQL lookup, typically milliseconds.
- ▸Data inlining. Small inserts can be stored directly in the catalog database and materialized to Parquet later, which suits frequent tiny writes that would degrade an Iceberg table.
- ▸Young ecosystem. The reference implementation is the DuckDB ducklake extension; third-party engine support is thin, and the spec iterated quickly through 0.x versions in 2025. Verify current stability and migration guarantees before committing production data.
- ▸The catalog is a dependency. Your Postgres now sits on the query path and needs the same HA and backup discipline as any OLTP database.
Which engines and vendors support each format in 2026?
Iceberg has the broadest neutral support: Spark, Flink, Trino, Snowflake, BigQuery, Redshift, ClickHouse, DuckDB and Databricks can all read it, and most can write it. Delta is strongest inside Databricks, Microsoft Fabric and Spark. DuckLake is DuckDB-first today, with interop paths to Iceberg rather than native third-party writers.
Details that change decisions:
- ▸Read support is not write support. Many engines read all three formats tolerably; the write path, meaning compaction, deletes and streaming upserts, is where maturity diverges. Verify writer capability per engine and version.
- ▸Catalogs are part of the ecosystem question. Iceberg has multiple independent open catalogs, including Apache Polaris, Lakekeeper and Nessie, plus Glue and Unity Catalog; Delta leans on Unity; DuckLake's catalog is the SQL database itself.
- ▸Translation layers exist. Apache XTable (incubating) converts metadata between Iceberg, Delta and Hudi, and UniForm bridges Delta to Iceberg readers. Both help read-side interop; neither makes dual-writing safe.
- ▸The market signal. Databricks acquiring Tabular, the company founded by Iceberg's creators, in 2024 pushed the industry toward Iceberg as the interchange layer even where Delta remains the internal format.
Which format minimizes lock-in?
Iceberg, on current evidence. It is Apache-governed, independently implemented by multiple competing vendors, and its REST catalog API decouples tables from any one platform. Delta is open source but effectively single-vendor in direction. DuckLake is fully open, yet has one dominant implementation, so practical portability remains narrower for now.
Lock-in hides in three places beyond the format itself:
- ▸The catalog. An open table behind a proprietary catalog with proprietary auth is not portable in practice. Prefer catalogs you can self-host and export, such as Polaris, Lakekeeper or Nessie, or verify export paths from managed ones.
- ▸Engine-specific features. Liquid clustering, vendor-only indexes and proprietary table properties quietly bind tables to one engine.
- ▸Egress and residency. A format is only as portable as your ability to move or re-home the data. Formats over object storage let the same tables run on MinIO or Ceph on-premises, exactly the flexibility argued for in Sovereign Cloud and EU Data Residency in 2026: An Engineering Playbook and priced out in When to Repatriate: A FinOps Decision Framework for Steady-State Workloads.
Cheap insurance: once a quarter, read your critical tables from a second engine and time the exercise. If the exit does not work in a drill, it does not exist.
How should you decide between Iceberg, Delta Lake and DuckLake?
Decide by engine gravity first, then team scale. Databricks or Fabric shops should default to Delta with UniForm enabled. Multi-engine estates running Trino, Snowflake, Spark or ClickHouse should default to Iceberg with an open REST catalog. Teams living in DuckDB and Postgres at sub-ten-terabyte scale can get real leverage from DuckLake today.
A compact rubric:
- ▸Choose Iceberg when more than one engine must write, you want vendor-neutral governance, or you are building a platform meant to outlive current tooling choices.
- ▸Choose Delta when Databricks or Fabric is the center of gravity and you want the least friction there, and enable UniForm so Iceberg readers are not locked out.
- ▸Choose DuckLake when the workload is analytics at small-to-mid scale, writes are frequent and small, and you already run Postgres. Its operational simplicity is a genuine feature, not a compromise.
- ▸Whatever you pick, feed it with idempotent ingestion, for example CDC streams as described in Change Data Capture in Practice: Debezium, Kafka, and the Outbox Pattern, and schedule table maintenance, meaning compaction, snapshot expiry and vacuum, from day one.
How TuniCyberLabs helps
TuniCyberLabs builds lakehouse and data platforms on EU-resident infrastructure: format and catalog selection, Iceberg and Delta deployments, CDC ingestion, DuckLake pilots for smaller estates, and migrations off proprietary warehouses, with security and data residency reviewed as part of the design rather than after it. Explore our cloud and data engineering services.