Delta Lake is what makes a pile of Parquet files behave like a database table. Understanding the mechanism — not just the API — is what lets you reason about concurrency, time travel, and why your table got slow.
A Delta table on disk is Parquet data files plus a _delta_log/ directory:
my_table/
├── part-00000-....snappy.parquet
├── part-00001-....snappy.parquet
└── _delta_log/
├── 00000000000000000000.json ← commit 0
├── 00000000000000000001.json ← commit 1
├── 00000000000000000002.json
└── 00000000000000000010.checkpoint.parquet
Each numbered JSON file is one atomic commit containing actions: add this file, remove that file, change this metadata. The current state of the table is the result of replaying those commits in order.
Two things follow immediately, and they explain most Delta behavior:
Deletes and updates don't edit Parquet files. Parquet is immutable. A DELETE writes a new
Parquet file without the deleted rows, and commits "remove old file, add new file." The old file
stays on disk until vacuumed — which is exactly why time travel works.
Readers never see partial writes. A reader resolves the table state from the log. A commit that hasn't landed is invisible; a commit that has landed is complete. That's the atomicity guarantee, and it's achieved without any lock on the data files.
Every tenth commit or so, Delta writes a checkpoint — a Parquet summary of accumulated state — so readers don't replay thousands of JSONs from zero.
Primary source: Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores (VLDB 2020). For a full free book: Delta Lake: The Definitive Guide (O'Reilly, 2024/2025) — no email gate.
Because old files linger until vacuumed, you can read the past:
SELECT * FROM sales VERSION AS OF 42;
SELECT * FROM sales TIMESTAMP AS OF '2026-07-01';
DESCRIBE HISTORY sales;
RESTORE TABLE sales TO VERSION AS OF 42;
DESCRIBE HISTORY is one of the most useful debugging commands on the platform: who wrote, when,
which operation, how many rows.
The limit people forget: time travel only reaches back as far as your retention window. VACUUM
physically deletes unreferenced files (default retention 7 days). After vacuuming, older versions
are gone — the log entry may exist, but the data files don't. Time travel is not a backup.
Delta uses optimistic concurrency control. Writers don't lock. Each assumes it will succeed, prepares its commit, then atomically claims the next log number. If two writers race for the same number, one wins; the loser re-reads and retries.
The practical consequence: concurrent appends to different parts of a table generally succeed;
concurrent writes touching the same files conflict. When you see ConcurrentAppendException in
production, that's this mechanism, and the fix is usually partitioning or clustering the writes so
they stop overlapping — not adding retries.
Small files are the classic killer: thousands of tiny Parquet files mean enormous per-file overhead.
The old way (know it, you'll see it): PARTITIONED BY on a low-cardinality column, plus
OPTIMIZE with ZORDER BY. Partitioning is easy to get wrong — partition on something
high-cardinality like a timestamp and you manufacture the small-file problem you were trying to fix.
The current way — Liquid Clustering:
CREATE TABLE sales (...) CLUSTER BY (customer_id, order_date);
-- or let Databricks choose and adapt the clustering keys for you
CREATE TABLE sales (...) CLUSTER BY AUTO;
Liquid clustering replaces both partitioning and ZORDER. You can change clustering keys without rewriting the table, which fixed partitioning's worst property: being nearly irreversible.
Verified status as of 2026-07-30 (clustering docs, updated 2026-07-10):
| Status | |
|---|---|
| Liquid clustering for Delta | GA, DBR 15.4 LTS+ |
| Liquid clustering for Iceberg | Public Preview, DBR 16.4 LTS+ |
CLUSTER BY AUTO |
GA for UC managed Delta, DBR 15.4 LTS+ |
| Converting a partitioned table → liquid clustering | GA (2026-06-12) |
Predictive Optimization runs OPTIMIZE and VACUUM automatically. It's GA, on by default for
accounts created on or after 2024-11-11, with rollout to older accounts completing August 2026.
It applies to Unity Catalog managed tables only — another reason to prefer managed
(source, 2026-07-28).
Apache Iceberg is the other major open table format. This used to be an either/or decision. It mostly isn't any more.
As of DAIS 2026, Unity Catalog managed Iceberg tables are GA, and the Iceberg REST Catalog API is GA for both read and write from external engines (What's new in Unity Catalog, 2026-06-16). Iceberg v3 and geospatial types are GA. Foreign Iceberg tables from Glue, Hive metastore, or Snowflake are GA read-only.
For an exam and for most interviews: know that Delta is the native default, that Iceberg is a first-class citizen now rather than a competitor to migrate to, and that the REST catalog is how outside engines get in.
DELETE FROM t WHERE id = 5 — what physically happens on storage?VACUUM with default retention yesterday. Can you time travel to a version from a month
ago?PARTITIONED BY over CLUSTER BY today?ConcurrentAppendException.CLUSTER BY._delta_log/. List the directory, open one JSON commit, read the add/remove
actions aloud. The abstraction becomes concrete in about ninety seconds, and this is the moment
most people actually "get" Delta.DESCRIBE HISTORY, then time travel back and show it returning.
Then VACUUM and show the same query now failing. The failure is the lesson.