← All modules

E1 — Data Ingestion and Loading

Why this module matters

21% of the Associate exam — the second-largest section, and the largest one that's purely mechanical skill rather than judgment. It's also where most production incidents are born. A badly built ingestion layer doesn't fail loudly; it silently drops columns, duplicates rows after a restart, or quietly costs ten times what it should in storage API calls.

The good news: Databricks has converged on a small number of correct patterns. Learn those and most of the difficulty evaporates.

The one idea to hold onto

Databricks now frames ingestion as three layers, and tells you to start at the most managed one and only drop down when it can't do the job:

  MOST MANAGED                                          LEAST MANAGED
  ┌────────────────────┐  ┌──────────────────┐  ┌────────────────────────┐
  │ Managed Connectors │→ │ Lakeflow         │→ │ Structured Streaming   │
  │ (Lakeflow Connect) │  │ Pipelines        │  │ (Spark APIs)           │
  └────────────────────┘  └──────────────────┘  └────────────────────────┘
   auth, CDC, retries,     orchestration,        full control, full
   schema evolution,       monitoring, data      responsibility
   API maintenance         quality, errors

"Databricks recommends starting with the most managed layer. If it doesn't satisfy your requirements (for example, if it doesn't support your data source), drop down to the next layer." — Ingest data (2026-07-10)

Engineers reflexively start at the bottom because that's where the interesting code is. The recommended direction is top-down. That's both an exam answer and good professional advice.

What you'll be able to do

1. Choose the right ingestion layer and defend the choice.
2. Configure Auto Loader correctly — schema location, evolution mode, rescued data — and explain what each option does when things go wrong.
3. Pick between directory listing, classic file notification, and file events, on any of the three clouds.
4. Write incremental ingestion in SQL with read_files and CREATE STREAMING TABLE.
5. Implement CDC with AUTO CDC, producing correct SCD Type 1 and Type 2 output.
6. Explain exactly-once semantics for both Auto Loader and COPY INTO.

Lessons

Then: Cheat sheet · Lab · Quiz

A note on the docs

While researching this module I hit two genuine inconsistencies in Databricks' live documentation — the valid values for cloudFiles.schemaEvolutionMode, and whether the rescued-data option carries a cloudFiles. prefix. Both are flagged where they arise.

I'm telling you rather than smoothing over it, for two reasons. It'll save you an afternoon when you hit it yourself. And "the docs contradict themselves here, so I tested it" is a genuinely good interview answer.

Facts verified 2026-07-30 against the pages cited in each lesson.