← All modules

E2 — Data Transformation and Modeling

Why this module matters

22% of the Associate exam — the largest single section. E1 got data into the platform. E2 is everything that happens after: layering it, declaring it, validating it, reshaping it, and modeling it so a business analyst can answer a question without asking you first.

It's also the section where the platform has changed most in the last two years. The API you'll be tested on is from pyspark import pipelines as dp. Half the material on the internet still says import dlt. Every code block in this module was copied from a documentation page I fetched on 2026-07-31, and the page's own last-updated date is next to it.

The one idea to hold onto

Streaming tables and materialized views are not two flavors of the same thing. They are two different answers to one question: what happens when the source changes?

                    Does a source row ever get UPDATED or DELETED?
                                     │
              ┌──────────────────────┴──────────────────────┐
              │ NO — append only                            │ YES
              ▼                                             ▼
      ┌────────────────────┐                    ┌──────────────────────────┐
      │  STREAMING TABLE   │                    │   MATERIALIZED VIEW      │
      │  @dp.table         │                    │   @dp.materialized_view  │
      │  spark.readStream  │                    │   spark.read             │
      └────────────────────┘                    └──────────────────────────┘
       each row seen once                        always correct at update time
       cheap, low latency                        recomputes joins & aggregates
       joins DON'T recompute                     seconds/minutes, not ms

"Joins in streaming tables do not recompute when dimensions change. This characteristic can be good for 'fast-but-wrong' scenarios. If you want your view to always be correct, you might want to use a materialized view." — Streaming tables (2026-07-10)

"Fast-but-wrong" is Databricks' own phrase, in their own docs, about their own product. Learn that sentence. It answers more exam questions than any other single fact in this module.

What you'll be able to do

1. Place a dataset in bronze, silver, or gold and defend the choice — including the ingestion frequency, and therefore the cost.
2. Write a Lakeflow pipeline in current Python (@dp.table, @dp.materialized_view, dp.create_streaming_table, dp.append_flow) and its SQL equivalent, and explain what a flow is separately from what a table is.
3. Choose between a streaming table and a materialized view from the shape of the source, not from habit.
4. Write expectations in both languages, predict exactly which rows survive each ON VIOLATION action, and pull the pass/fail counts out of the pipeline event log with SQL.
5. Use window functions, higher-order functions, and VARIANT — and know the specific cost of reaching for a UDF instead.
6. Build a Kimball-style star schema on Databricks: surrogate keys, informational primary and foreign keys, and SCD Type 2 via AUTO CDC.

Lessons

Then: Cheat sheet · Lab · Quiz

A note on the docs

E1 flagged two live inconsistencies in Databricks' documentation. E2 found three, and one of them is a page that contradicts itself in the same scroll.

1. Do materialized views with expectations support incremental refresh? Three pages say no. One page — the newest — says yes. Flagged in lesson 3.
2. Do materialized views support generated columns? The limitations section of the pipeline CREATE MATERIALIZED VIEW reference says no. That same page's examples use them. Flagged in lesson 5.
3. What privilege does creating a streaming table need? The streaming table reference asks for CREATE MATERIALIZED VIEW on the schema. Flagged in lesson 2.

I'm surfacing these rather than picking a side silently. Two reasons, same as E1: it'll save you an afternoon, and "the docs disagreed so I tested it in a scratch schema" is a better answer in an interview than any fact I could hand you.

On the 22%

Verified. The Data Engineer Associate exam guide (version as of 2026-05-04) lists "Data Transformation and Modeling" as Section 3 at 22% — the largest of the seven sections. The exam is 45 scored questions in 90 minutes, so 22% is roughly 10 questions. This module is sized accordingly.

Everything else here traces to a docs.databricks.com or learn.microsoft.com/azure/databricks page listed in each lesson's frontmatter.

Facts verified 2026-07-31.

Keeps playing into the following modules — 527 min from here to the end of the course.