← All modules

E7 — Apache Spark Internals and Optimization

Why this module matters

This is not an exam section. Nothing here is scored on the Data Engineer Associate, and if you are three days from sitting it, go do E6 instead.

It matters anyway, for two reasons.

The first is that E6 teaches you which Databricks feature fixes a problem; E7 teaches you what the problem is. Those are different skills and the second one keeps working when you're on EMR, on Dataproc, on an on-prem YARN cluster, or reading a stack trace at two in the morning with no Spark UI available. Adaptive query execution, liquid clustering and Photon are excellent. They are also Databricks' implementations of ideas that live in the engine, and when they don't fire you need to know why.

The second is that this material is what separates "I ran a job" from "I run this platform" in an interview. Nobody asks a senior candidate to define OPTIMIZE. They ask why a shuffle creates a stage boundary, or what a broadcast hash join costs the driver.

Where E7 stops and E6 starts

Read this table before you start. The two modules are deliberately complementary and you will waste your time if you read them as duplicates.

Rule of thumb: E6 is the operator's module, E7 is the engineer's. Where a Databricks feature is the right answer, E7 points at E6 rather than re-explaining it.

The one idea to hold onto

A partition is a unit of work. Everything else follows from that.

   FILES / ROWS                    PARTITIONS                      TASKS
   ───────────                     ──────────                      ─────
   your data          ──split──→   the engine's unit    ──1:1──→   what runs
                                   of division                     on a core

   too few partitions  → idle cores, huge tasks, spill, OOM
   too many partitions → scheduling overhead swamps the work
   uneven partitions   → skew: one task holds the whole job open

Parallelism is not "how big is my cluster". Parallelism is the number of tasks in the current stage, capped by the number of task slots available. Databricks states the mapping plainly:

"Spark uses a 1:1 mapping between task slots and available cores." — Run multiple Structured Streaming queries on the same cluster (2026-07-10)

A stage with one task uses one core on a 128-core cluster. A stage with 200 tasks of wildly unequal size finishes when the biggest one does. Almost every performance problem in this module is a partition-shaped problem wearing a costume.

What you'll be able to do

1. Trace a query from action → job → stages → tasks and explain where each stage boundary came from.
2. Describe what physically happens in a shuffle — map-side write, local disk, fetch — and size shuffle partitions from first principles instead of by superstition.
3. Name Spark's join strategies, explain the selection rules and thresholds, and read a physical plan to see which one you got.
4. Induce skew deliberately, detect it, and fix it three different ways — knowing which one to reach for first.
5. Explain Catalyst's four phases, what predicate and projection pushdown do, what AQE re-optimizes at runtime, and — most usefully — what the optimizer cannot fix for you.
6. Explain the unified memory model, choose a storage level with a reason, and quantify why a Python UDF costs what it costs.

Lessons

Then: Cheat sheet · Lab · Quiz

Which Spark am I actually running?

Every config default in this module is copied from the Apache Spark 4.2.0 documentation (spark.apache.org/docs/latest as of 2026-07-31) or from a Databricks page, and labelled which. That matters, because the runtime you're on decides which set applies.

Source: Databricks Runtime release notes versions and compatibility (page last updated 2026-07-23)

⚠️ The Spark 4.2 Python change matters more than most. Arrow optimization for regular Python UDFs became the default in Spark 4.2 — so the same UDF has meaningfully different economics on DBR 19 than on DBR 18 LTS. Lesson 5 has the exact quotes.

A note on the docs

Researching this module surfaced three things worth flagging up front, all detailed in the lessons:

1. "Narrow" and "wide transformation" are not defined in the current Apache Spark programming guides. The vocabulary is real and universally used, but its documented home is the Scala API's dependency classes, not the tuning docs. Lesson 1 quotes the actual source.
2. cache() does not mean the same thing for a DataFrame as for an RDD, and the storage-level table in the RDD programming guide doesn't list the level DataFrames actually use. Lesson 5.
3. The adaptive broadcast threshold on Databricks is not the Apache Spark one, and it's a different property name. Lesson 3.

I could not verify the default of spark.sql.join.preferSortMergeJoin from any current Apache Spark documentation page — it's referenced in passing on the performance-tuning page but not documented there. Flagged in lesson 3 rather than guessed.

Facts verified 2026-07-31 against the pages cited in each lesson.

▶ Play this module · 47 min

Keeps playing into the following modules — 187 min from here to the end of the course.