Delta Lake Deep Dive

What Delta adds over Parquet

A Delta table is Parquet data files plus a transaction log. The Parquet files are inert; the log is the table. That one addition buys you: ACID transactions on object storage, scalable metadata (no more listing a million files to plan a query), time travel, schema enforcement and evolution, UPDATE/DELETE/MERGE as first-class DML, streaming-and-batch unification on the same table, and an audit trail via DESCRIBE HISTORY. When an interviewer asks "why not just Parquet?", the crisp answer is: Parquet is a file format, Delta is a table format — it defines which files constitute a consistent version of the table at a point in time, and lets concurrent readers and writers agree on that without locks.

The _delta_log transaction log

Every Delta table directory contains a _delta_log/ folder. Each commit is a JSON file named by a zero-padded, monotonically increasing version: 00000000000000000000.json, ...0001.json, and so on. A commit holds actions:

• add — a data file now part of the table, with partition values, size, and per-file column stats (min/max/null counts, used for data skipping).
• remove — a file logically deleted from the table (the physical file lingers until VACUUM).
• metaData — schema, partition columns, table properties.
• protocol — minimum reader/writer versions, and on modern tables the explicit list of table features (deletion vectors, change data feed, liquid clustering, type widening, etc.). A client that doesn't support a required feature must refuse to read or write — that is how Delta stays safe across engine versions.
• commitInfo — operation, user, timestamps; what DESCRIBE HISTORY surfaces.

Replaying every JSON file from version 0 would get slow, so every 10 commits (by default) Delta writes a checkpoint — a Parquet snapshot of the full table state at that version, referenced from _last_checkpoint. A reader loads the latest checkpoint, applies the few JSON commits after it, and has the current state. Know this mechanism cold: "walk me through what happens in _delta_log when I run an UPDATE" is a classic warm-up question, and the answer (new Parquet files written, an atomic commit with remove + add actions — or a deletion-vector update, see below) shows you actually understand the format.

ACID via optimistic concurrency

Delta gets atomicity from the atomic creation of the next commit file (put-if-absent on the version number — on Unity Catalog managed tables this is brokered by a commit coordinator). Concurrency is optimistic: writers don't take locks. Each writer records the table version it read, does its work, then attempts to commit. If another writer committed in the meantime, Delta checks whether the two operations actually touched the same data:

• No conflict: two appends (blind INSERTs) to the same table — both succeed; the loser is logically reordered after the winner.
• ConcurrentAppendException: your MERGE/UPDATE/DELETE read files that a concurrent operation added in a region your predicate could match.
• ConcurrentDeleteReadException / ConcurrentDeleteDeleteException: a file you read or planned to remove was removed by someone else (often a concurrent OPTIMIZE or another MERGE).
• MetadataChangedException / ProtocolChangedException: schema or protocol changed underneath you.

The practical lever: tight, disjoint predicates. Two MERGEs into the same table can run concurrently if their conditions provably touch disjoint partitions or clustered ranges; two MERGEs with broad predicates will fight. Retrying the loser is normal and safe — the operation re-reads the new snapshot.

Schema enforcement and evolution

Delta validates every write against the table schema: extra columns, missing non-nullable columns, or incompatible types fail the write instead of silently corrupting the table. Evolution is opt-in:

• mergeSchema (option on a write, or spark.databricks.delta.schema.autoMerge.enabled for MERGE) — new columns are appended to the schema; existing rows read them as null.
• overwriteSchema with mode("overwrite") — replaces the schema entirely; destructive, use deliberately.
• Type widening (a table feature) — lets columns widen safely, e.g. INT to BIGINT, without rewriting the table. Narrowing or incompatible changes still require an explicit rewrite.

Time travel and retention

Because the log keeps every version, you can query the past:

-- SQL
SELECT * FROM finance.silver.gl_balances VERSION AS OF 412;
SELECT * FROM finance.silver.gl_balances TIMESTAMP AS OF '2026-06-01';

-- PySpark
df = (spark.read.format("delta")
      .option("versionAsOf", 412)
      .table("finance.silver.gl_balances"))

-- Recover from a bad write
RESTORE TABLE finance.silver.gl_balances TO VERSION AS OF 411;

Time travel is only as deep as two retention settings allow: delta.logRetentionDuration (default 30 days of log entries) and delta.deletedFileRetentionDuration (default 7 days before VACUUM may physically delete removed files). A version is reconstructable only if both its log entries and its data files still exist. In practice the binding constraint is VACUUM.

VACUUM: the retention vs time-travel trade-off

VACUUM finance.silver.gl_balances;            -- default 7-day threshold
VACUUM finance.silver.gl_balances RETAIN 168 HOURS;

VACUUM physically deletes files no longer referenced by any version newer than the retention threshold. The trade-off is direct: longer retention = deeper time travel and safer streaming restarts, but more storage cost; shorter retention = cheaper storage, but you lose rollback depth. Two rules of thumb: never set retention below your longest-running query or streaming trigger interval (readers of an old snapshot will hit FileNotFoundException mid-query), and treat overriding the 7-day safety check (spark.databricks.delta.retentionDurationCheck.enabled = false) as a red flag in code review. Note that on Unity Catalog managed tables, predictive optimization can run VACUUM (and OPTIMIZE) for you — see below.

OPTIMIZE and the small-files problem

Streaming ingestion, frequent small batches, and per-partition writes all produce thousands of small files. Each file costs an open/read round-trip and a row in the metadata scan, so query latency degrades even when total data volume is modest. OPTIMIZE bin-packs small files into larger ones (target ~1 GB by default, auto-tuned by the platform):

OPTIMIZE finance.silver.gl_balances;
OPTIMIZE finance.silver.gl_balances WHERE fiscal_period = '2026-05';

OPTIMIZE is idempotent and transactional — it only rewrites layout, never data, so readers are unaffected and a concurrent failure just retries. Complement it with optimizeWrite and autoCompact table properties so files are born reasonably sized in the first place, and you run full OPTIMIZE less often.

Z-ORDER vs liquid clustering

Data skipping works because each file's stats record min/max per column; a filter can skip files whose range can't match. Layout determines how effective that is. Three generations of answer:

-- Legacy layout (know it; don't choose it for new tables)
OPTIMIZE finance.silver.gl_balances ZORDER BY (company_code, account_id);

-- Current guidance: liquid clustering
CREATE TABLE finance.silver.gl_balances (...)
CLUSTER BY (company_code, account_id);

ALTER TABLE finance.silver.gl_balances CLUSTER BY (company_code, fiscal_period);

-- Preferred default as of mid-2026: let the platform pick and evolve keys
CREATE TABLE finance.silver.gl_balances (...) CLUSTER BY AUTO;

Current guidance (as of mid-2026): Databricks recommends liquid clustering for all new Delta and Iceberg tables, preferably automatic liquid clustering via CLUSTER BY AUTO, where key selection is powered by predictive optimization based on actual query patterns. Partitioning + Z-ORDER is now explicitly legacy guidance, and liquid clustering cannot coexist with either on the same table. In interviews, show you know both worlds: you tuned real tables with partitioning and Z-ORDER, and you know the migration path forward (ALTER TABLE ... CLUSTER BY on new tables, or recreate-and-backfill for old ones).

Deletion vectors: merge-on-read DML

Classically, deleting one row meant rewriting the whole Parquet file containing it (copy-on-write). With deletion vectors enabled, DELETE/UPDATE/MERGE instead write a compact bitmap marking rows as removed; the data file stays put and readers apply the mask (merge-on-read). Writes get dramatically cheaper for point-DML; reads pay a small masking cost until OPTIMIZE or a later rewrite purges the vectors. Photon, the vectorized native engine, evaluates deletion-vector masks efficiently, which is a large part of why the read penalty is acceptable — on non-Photon or old-runtime clusters the read cost is more visible. The workspace default for auto-enabling deletion vectors is shifting from Disabled to "All new tables" as of mid-2026, so assume new tables have them. Caveat for the architecture conversation: deletion vectors are a writer/reader table feature, so external engines and older clients must support it, and REORG TABLE ... APPLY (PURGE) exists to materialize deletes back into plain Parquet when you need to drop the feature or hand files to a non-Delta reader.

Change Data Feed

CDF makes a Delta table emit row-level changes — inserts, updates (pre- and post-image), deletes — so downstream consumers can process deltas of the Delta table instead of full snapshots:

ALTER TABLE finance.silver.gl_balances
  SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

-- Batch read of a change window
SELECT * FROM table_changes('finance.silver.gl_balances', 412, 418);

-- Streaming read (PySpark)
changes = (spark.readStream.format("delta")
           .option("readChangeFeed", "true")
           .option("startingVersion", 412)
           .table("finance.silver.gl_balances"))

Each change row carries _change_type (insert, update_preimage, update_postimage, delete), _commit_version, and _commit_timestamp. The canonical use: a Silver table maintained by MERGE feeds a Gold aggregate — without CDF the Gold job re-reads everything; with CDF it consumes only changed keys and applies a targeted MERGE. CDF data lives under _change_data and obeys the same retention as time travel, so a consumer that falls behind longer than retention loses its window — design your lag alerts accordingly. CDF is forward-only from the moment you enable it; it cannot backfill history.

MERGE INTO and idempotent upserts

MERGE INTO finance.silver.gl_balances AS t
USING staged_gl_extract AS s
  ON  t.company_code = s.company_code
  AND t.account_id   = s.account_id
  AND t.fiscal_period = s.fiscal_period
WHEN MATCHED AND s.op = 'D' THEN DELETE
WHEN MATCHED AND s._loaded_at > t._loaded_at THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Internally MERGE is two passes: an inner join between source and target to find which target files contain matches, then a rewrite of just those files (or, with deletion vectors, a much cheaper masked update) plus appends for new rows — all in one atomic commit. Three things make a MERGE production-grade:

• Deterministic source: deduplicate the staged batch on the merge key first (latest record per key). Duplicate keys in the source make MERGE fail outright with a multiple-match error.
• Idempotency: a re-run of the same batch must be a no-op. Key on business keys, guard updates with a watermark comparison (the _loaded_at check above), and the same file replayed twice converges to the same state.
• Pruning in the ON clause: add partition/cluster-key predicates so MERGE scans candidate files, not the whole table — this is also what keeps concurrent MERGEs from conflicting.

Managed vs external tables

The 2026 default answer is managed: Unity Catalog managed tables get predictive optimization and automatic clustering, and UC removes the old "managed = locked in DBFS root" objection. External tables remain right when other engines write the files, when storage lifecycle and cost controls are owned by a platform team at the storage-account level, or when you must guarantee that dropping catalog objects can never delete data.

UniForm and Iceberg interoperability

Delta and Iceberg both sit on Parquet, so interop is a metadata problem. UniForm makes a Delta table generate Iceberg metadata alongside the Delta log, so Iceberg clients (Snowflake, Trino, etc.) can read the table — read-only from the Iceberg side; Delta remains the writer. Separately, Unity Catalog now offers managed Apache Iceberg tables (GA as of 2026) with full read/write through UC's Iceberg REST Catalog, and Iceberg v3 itself adopted deletion vectors and row tracking — the formats are visibly converging. For interviews, the one-liner: UniForm = Delta table with an Iceberg read surface; UC managed Iceberg = a true Iceberg table governed by Unity Catalog. Choose based on which engine owns writes.

Predictive optimization

Predictive optimization lets Databricks decide when to run OPTIMIZE, VACUUM, and ANALYZE on Unity Catalog managed tables, based on observed access patterns — it is enabled by default for accounts created on or after November 11, 2024 and has been rolling out broadly since. It also powers the key selection behind CLUSTER BY AUTO. Implication for how you talk about maintenance: the hand-tuned OPTIMIZE/VACUUM job is becoming a legacy-table and external-table concern; on managed tables your job is to verify it's on, watch the spend it incurs, and override only with cause. Check the current docs for exact scope before quoting specifics in an interview — the covered operations have been expanding release by release.

Self-check