← All modules

E6 — Troubleshooting, Monitoring, and Optimization

Why this module matters

10% of the Associate exam — small on paper, and the section that most separates people who have run something in production from people who have only built it.

The reason is that every other module teaches you to make a thing. This one teaches you to answer "why is the thing slow, expensive, or broken?" — and that question has a specific, evidence-driven method attached to it. Most engineers don't use the method. They guess: bump the cluster, add a repartition, sprinkle cache(), re-run and hope. Sometimes it works, which is the worst possible outcome, because it teaches the wrong lesson.

The exam guide's own sample question for this section is a trap built exactly on that instinct. It describes a stage where most tasks finish in 30 seconds and one runs over 10 minutes, with max shuffle read of 5 GB against a median near 400 MB. Three of the four options are plausible-sounding tuning moves. The correct answer is to confirm that the automatic mechanism which already handles this is switched on.

"A data engineer notices a batch job's duration has doubled after a new data source was onboarded. In the Spark UI, the longest stage shows that most tasks finish in under 30 seconds, but one task takes over 10 minutes. […] B. Confirm adaptive query execution with skew join handling is active to automatically split the oversized partition at runtime" — DE Associate Exam Guide, May 4 2026, Sample Question 1, answer B

Learn the method and you'll answer these without thinking hard.

The one idea to hold onto

Measure, localise, then fix — and the platform has already fixed most of it.

  1. WHICH JOB?        Jobs Timeline        → gaps, failures, one long job vs many tiny ones
          ↓
  2. WHICH STAGE?      Stage list           → sort by duration; read Input/Output/Shuffle columns
          ↓
  3. WHAT'S WRONG?     Stage detail page    → skew? spill? one task? I/O bound?
          ↓
  4. WHAT FIXES IT?    Usually a setting    → AQE, Photon, liquid clustering, predictive optimization
                       you already have,
                       not code you write

That last box is the part people resist. Databricks has spent a decade turning hand-tuning into defaults. Adaptive query execution splits skewed partitions automatically. Predictive optimization runs OPTIMIZE and VACUUM for you. Liquid clustering changes keys without a rewrite. Photon makes scans and joins faster with no code change.

So the professional move is usually to verify a mechanism is enabled, not to write clever code. Salting a join key by hand is a 2019 answer. It is still occasionally correct — and knowing when is the actual skill.

What you'll be able to do

1. Walk the Spark UI in order — timeline, stage, task metrics — and name the bottleneck with evidence.
2. Recognise skew, spill, shuffle cost, and small files from stage-level numbers, and apply the specific fix for each.
3. Explain liquid clustering, CLUSTER BY AUTO, OPTIMIZE, VACUUM, deletion vectors, and predictive optimization — including which ones apply to which table types.
4. Say precisely what Photon accelerates and, more importantly, what it does not.
5. Read a query profile and a pipeline event log, and query the system tables that answer "what is this costing us and is it getting slower?"
6. Diagnose cluster startup failures, library conflicts, and out-of-memory errors without guessing.

Lessons

Then: Cheat sheet · Lab · Quiz

What the exam actually asks

The five published objectives, verbatim from the May 4 2026 exam guide:

• "Identify trends in job performance using the Lakeflow Jobs run history view to compare current execution times against historical baselines."
• "Use the Lakeflow Jobs UI to monitor pipeline health by interpreting job statuses, viewing DAG-based task graphs to spot upstream blockers, and tracking pipeline run times and failure rates."
• "Identify common performance bottlenecks such as data skew, shuffling, and disk spilling by interpreting stage-level metrics in the Spark UI."
• "Understand the features of Liquid Clustering and predictive optimization."
• "Diagnose cluster startup failures, library conflicts, and out-of-memory issues."

Note what's not there: no partition tuning theory, no memory-management internals, no Spark scheduler design. This section is operational, not architectural. Lessons 1, 2 and 5 map to it most directly; lesson 3 covers the Liquid Clustering / predictive optimization objective; lesson 4 is partly beyond the objectives but is where the real-world money is.

A practice resource worth your time

databricks-academy/spark-ui-simulator is a live, Databricks-Academy-maintained GitHub repository containing 30+ educational experiments built on pre-recorded Spark job data. You open a local index.html and click through frozen Spark UI captures for jobs with deliberately induced problems.

Verified from the GitHub API on 2026-07-31: created 2025-08-18, last pushed 2025-10-23, not archived. GitHub reports its license as "Other" (NOASSERTION) — check the LICENSE file yourself before reusing material commercially.

It is genuinely the fastest way to build Spark UI pattern recognition without breaking a real cluster. Lesson 1 tells you how to use it.

A note on the docs

Researching this module surfaced one real contradiction in Databricks' live documentation about which runtime version OPTIMIZE … FULL requires, and one place where I could not find a definitive statement about the default enablement of deletion vectors. Both are flagged where they arise, in lesson 3.

There is also a stale term in the exam guide itself — Sample Question 4's keyed answer uses "high-concurrency cluster", a name Databricks retired in favour of access modes. Flagged in lesson 5. Learn the current names; recognise the old ones on the exam.

Facts verified 2026-07-31 against the pages cited in each lesson.

▶ Play this module · 66 min

Keeps playing into the following modules — 253 min from here to the end of the course.