Before 2020 the standard enterprise setup was two systems glued together.
A data lake held everything — cheap object storage, open file formats, any kind of data, scale to petabytes. It was where data landed. What it lacked was transactions. Two writers at once could corrupt a table. A reader could see a half-written result. There was no schema enforcement, so the lake slowly became a swamp.
A data warehouse sat downstream. It had transactions, schemas, fast SQL, and BI tool support. What it lacked was openness and reach — proprietary storage, expensive to scale, and effectively useless for machine learning, which wants raw files and Python, not JDBC and SQL.
So companies ran both, and paid a tax: every dataset was stored twice, ETL jobs copied between them continuously, and the two copies drifted. The warehouse was authoritative for BI; the lake was authoritative for ML; nobody could say which was authoritative for the truth.
The lakehouse proposal — argued formally in the 2021 CIDR paper by Armbrust, Ghodsi, Xin, and Zaharia — was to stop copying and instead add the warehouse's missing guarantees directly on top of open files in object storage. One copy of the data. Open format. ACID transactions. Fast enough for BI, open enough for ML.
Read the primary source. The paper is nine pages and unusually readable: Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics (CIDR 2021). If you only read one thing outside this course, read this.
That's the architecture. Databricks is one implementation of it. Keep the two ideas separate in your head — "lakehouse" is a pattern, and it's implementable on other stacks too. Interviewers ask this.
Here is the part that pays off for the rest of the course. A Databricks deployment is split into three pieces, and who owns which piece is the whole game.
┌──────────────────────────────────────────────────────────┐
│ CONTROL PLANE (Databricks account) │
│ Web UI · REST APIs · job scheduler · query routing │
│ notebook source · cluster manager · Unity Catalog │
└──────────────────────────────────────────────────────────┘
│ │
▼ ▼
┌──────────────────────────┐ ┌─────────────────────────────┐
│ SERVERLESS COMPUTE PLANE │ │ CLASSIC COMPUTE PLANE │
│ (Databricks account) │ │ (YOUR cloud account) │
│ instant-start pooled VMs │ │ VMs in your VNet/VPC │
│ Databricks-managed net │ │ you own network + firewall │
└──────────────────────────┘ └─────────────────────────────┘
│ │
└──────────────┬───────────────┘
▼
┌──────────────────────────────┐
│ STORAGE (YOUR cloud acct) │
│ ADLS Gen2 / S3 / GCS │
│ your data never moves here │
└──────────────────────────────┘
Control plane — lives in the Databricks cloud account. It's the brain: the web application, the REST API, the job scheduler, cluster lifecycle management, and notebook source code. Unity Catalog's metadata lives here too.
Classic compute plane — VMs that run in your own cloud subscription, inside your VNet (Azure) or VPC (AWS/GCP). You own the network, the routing, the firewall rules, the private endpoints. This is what you pick when the security team needs full control of egress or you need custom networking.
Serverless compute plane — VMs that run in the Databricks account, pre-warmed and pooled. They start in seconds instead of minutes. You give up direct network ownership in exchange for speed and no idle cost. Network control moves to account-level constructs like Network Connectivity Configurations rather than your own VNet rules.
Storage is separate from all three and is always yours. Your data sits in your ADLS Gen2 account, your S3 bucket, your GCS bucket. Compute reaches into it. This is the single most important consequence of the design: compute and storage are decoupled, so you can scale, kill, or resize compute without touching data, and multiple engines can read the same table.
Source: High-level architecture — Databricks' own page naming these three components.
Trace a failure through it and the value becomes obvious. "My job can't read the source table."
Engineers who don't hold this model guess randomly between those four. Engineers who do, bisect.
One more distinction that trips people up constantly, and it's worth burning in now.
| Managed table | External table | |
|---|---|---|
| Who picks the storage path | Unity Catalog | You |
DROP TABLE |
deletes data | leaves data |
| Predictive Optimization | ✅ eligible | ❌ not eligible |
| Best default? | yes | only when you must |
Managed tables live in a storage location Unity Catalog controls, and they're the only ones eligible
for Predictive Optimization — Databricks automatically running OPTIMIZE and VACUUM for you.
External tables point at a path you specify; you keep the layout, and you keep the maintenance
burden.
Default to managed. Choose external when another engine owns the data's lifecycle, when you're mid-migration, or when a contractual requirement fixes the physical path.
Source: Predictive optimization (page updated 2026-07-28) — confirms it applies to Unity Catalog managed Delta and Iceberg tables only.
Databricks publishes a Well-Architected Lakehouse framework, deliberately echoing the cloud providers' equivalents. Five pillars are the industry-standard ones; two are specific to Databricks.
Shared with AWS/Azure/GCP frameworks:
Databricks-specific: 6. Data and AI governance 7. Interoperability and usability
Those last two are the ones to remember, because they're the ones an interviewer can use to tell whether you've actually read the framework. Module A1 in the architecture track takes all seven apart properly.
Source: Well-architected lakehouse · Azure mirror: learn.microsoft.com