A0 — Lakehouse Architecture Foundations

Why this module matters

E0 lesson 1 gave you the lakehouse in three paragraphs: two systems, a tax, one copy instead. That's enough to build pipelines. It is nowhere near enough to sit in a design review.

The difference between an engineer and an architect on this topic is not vocabulary. It's the ability to answer the hostile questions:

"Why is this better than just buying Snowflake?"
"You said 'open format'. Open to whom? What can I actually do with that openness that I can't do today?"
"What does this architecture give up? Don't tell me nothing."
"Our platform team wants Iceberg, the incumbent is Delta. Does this decision matter, and how much will it cost me to be wrong?"
"Where does the lakehouse fall over?"

An architect who can only recite the upside is worse than useless in that room, because they will lose the argument to the first person who has read a benchmark.

So this module is built on the primary literature — the CIDR 2021 lakehouse paper, the VLDB 2020 Delta Lake paper, the SIGMOD 2022 Photon paper — plus two independent-minded academic papers that push back, and the current Databricks documentation for what actually shipped. Every number and every quote comes from a document that was fetched and read.

The one idea to hold onto

The lakehouse is one specific bet, stated plainly by its own authors:

"The key question is whether one can combine these benefits in an effective way: in particular, Lakehouses' support for direct access means that they give up some aspects of data independence, which has been a cornerstone of relational DBMS design." — Lakehouse (CIDR 2021), §3

Everything else follows from that sentence. The lakehouse publishes its storage format as part of its public API so that any engine can read it. That is the entire value proposition — and it is simultaneously the source of every serious criticism the architecture faces. Openness is the constraint.

Hold both halves. If you only hold the first, you're marketing. If you only hold the second, you're a contrarian with no plan.

What you'll be able to do

Reconstruct the argument for the lakehouse from first principles — including what problem the warehouse solved, what problem the lake solved, and why the two-tier architecture became the default before it became the target.
Explain how ACID lands on an object store: the transaction log, the commit protocol, and the per-cloud atomicity primitives — and why that design caps write throughput.
Name and apply the three format-independent performance levers, and say which one you'd reach for given a specific workload.
Compare Delta, Iceberg, and Hudi on design, not on marketing — and state a defensible format strategy for a real organisation.
Explain why Photon is written in C++ and uses interpreted vectorization rather than code generation, and predict which of your workloads it will and won't accelerate.
Critique medallion, data mesh, and lakehouse federation — and identify the workloads where the lakehouse is the wrong answer.

Lessons

#	Lesson	Read	Listen
1	The architectural argument	30 min	12 min
2	Warehouse guarantees on object storage	34 min	13 min
3	Open table formats and the convergence	32 min	12 min
4	The engine layer	28 min	12 min
5	Patterns and anti-patterns	31 min	12 min

Then: Cheat sheet · Design exercise · Quiz

The primary sources

Read these. They are short, they are well written, and quoting them accurately is a superpower in a design review.

Paper	Venue	What it's for
Lakehouse: A New Generation of Open Platforms…	CIDR 2021	The argument. 9 pages.
Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	VLDB 2020	The mechanism.
Photon: A Fast Query Engine for Lakehouse Systems	SIGMOD 2022	The engine.
Analyzing and Comparing Lakehouse Storage Systems	CIDR 2023	Delta vs Hudi vs Iceberg, measured.
Active Data Lakes: Regaining Physical Data Independence…	VLDB 2026	The best critique. Not from a vendor.

⚠️ Three of those five have Databricks authors. The CIDR 2021, VLDB 2020 and SIGMOD 2022 papers are all Databricks-authored; the CIDR 2023 comparison paper has a Databricks co-author (Tathagata Das) and Matei Zaharia on it. That does not make them wrong — they're peer-reviewed and their methods are stated — but you should read them the way you'd read any paper where the authors sell the thing being measured. Lessons 1 and 3 point out specifically where the framing is generous.

The TU Munich paper (Ginter & Leis) is the one with no dog in the fight, and it's the one that will make you a better architect.

A note on what changed after the papers were written

Several load-bearing facts in the 2020–2021 papers are now historically interesting rather than operationally true. This module flags each one where it arises, because quoting a superseded fact in a design review is how you lose the room:

S3 eventual consistency (Delta paper §2.2) — S3 now provides strong read-after-write consistency.
S3 has no put-if-absent (Delta paper §3.2.2) — S3 now supports conditional writes via If-None-Match.
Delta has no merge-on-read (CIDR 2023 paper §2.3) — deletion vectors are GA.
Hudi does not support concurrent writers (CIDR 2021 §3.2, Delta paper §8) — by CIDR 2023, Hudi is described as using optimistic concurrency control with table-level locks, and its current documentation lists concurrency control as a feature.

Facts verified 2026-07-31 against the pages cited in each lesson.

Keeps playing into the following modules — 140 min from here to the end of the course.