← All modules

E3 — Working with Lakeflow Jobs

Why this module matters

16% of the Associate exam — roughly 7 of the 45 scored questions. But the reason to take it seriously isn't the weighting. It's that orchestration is the layer where every other layer's failures become your pager.

Ingestion breaks at 3am. Somebody has to decide whether it retries, whether it alerts, whether the downstream transform runs anyway, and whether re-running it duplicates a day of revenue. Those are all Lakeflow Jobs decisions, and most of them are made once, badly, at job-creation time, and never revisited.

The exam guide is unusually specific about what it wants here (DE Associate Exam Guide, exam version as of May 4, 2026):

Section 4: Working with Lakeflow Jobs (16%)

• Implement control flows (retries and conditional tasks such as branching and looping) using Lakeflow Jobs for pipeline orchestration
• Configure common tasks (notebook, SQL query, dashboard, and pipeline tasks) and their dependencies using Lakeflow Jobs and its DAG‑based task graph
• Implement job schedules using Lakeflow Jobs with an understanding of trigger types (scheduled, file arrival, and table update)
• Choose between time‑based and data‑driven triggers based on data availability and pipeline dependencies

Note the fourth bullet. It's a judgment objective, not a recall objective. That's the one people lose points on.

The name

The product is Lakeflow Jobs. It was renamed on June 11, 2025:

"The product known as Databricks Jobs is now Lakeflow Jobs. No migration is required to use Lakeflow Jobs." — June 2025 release notes (page updated 2026-07-27)

"Workflows" is dead as a product name — the Workflows and Pipelines items were removed from the left navigation on June 18, 2025 and replaced by a single Jobs & Pipelines entry. The system tables schema moved too: system.workflow is now system.lakeflow.

If your study material says "Databricks Workflows", it predates June 2025. Assume everything else in it is equally stale.

The one idea to hold onto

A job is a DAG of tasks, and every interesting decision is about what happens at the edges.

        ┌──────────┐
        │ ingest   │
        └────┬─────┘
             │  ← the edge carries a Run if condition
        ┌────▼─────┐
        │ if/else  │  ← and a branch label (true / false)
        └──┬────┬──┘
     true  │    │  false
     ┌─────▼┐  ┌▼──────────┐
     │ load │  │ quarantine│
     └──────┘  └───────────┘

Tasks are the boring part — a notebook is a notebook. The exam, and production, both live on the edges: dependencies, Run if conditions, branch labels, task values flowing forward, and what the run status is when half the graph didn't execute.

What you'll be able to do

1. Name every task type and pick the right one, including when to use Run Job instead of adding a task.
2. Choose between the six trigger types on evidence, and configure cron, file arrival, and table update triggers correctly — including the rate-control options nobody sets.
3. Build branching and looping with If/else and For each, pass values between tasks, and explain why 12.0 == 12 is false.
4. Configure retries, timeouts, notifications and queueing so failures are loud, bounded, and safe to repair.
5. Query system.lakeflow.* joined to system.billing.usage to answer "what does this job cost and why did it fail last Tuesday."

Lessons

Then: Cheat sheet · Lab · Quiz

A note on the docs

Researching this module turned up four genuine problems in Databricks' live documentation, all flagged where they arise:

1. Three different vocabularies for the same run outcome — the UI's status names, the {{tasks.<name>.result_state}} values, and the system tables' result_state values. They don't match, and one of them spells "cancelled" with two Ls. (Lessons 4 and 5.)
2. Three different vocabularies for trigger types too — UI names, {{job.trigger.type}} values, and system table trigger_type values. The system tables distinguish CRON from PERIODIC; the dynamic reference list doesn't. (Lessons 2 and 5.)
3. A flat contradiction on the retry default. The notifications page says tasks are retried three times by default. The task configuration page says most configurations don't retry at all. (Lesson 4.)
4. A contradiction inside the continuous-jobs page itself about whether retry policies work. (Lessons 2 and 4.)

There's also a behavior change landing in early August 2026 that breaks existing continuous-job automation. That's days away from this module's verification date, so lesson 2 covers it explicitly.

Facts verified 2026-07-31 against the pages cited in each lesson.

▶ Play this module · 78 min

Keeps playing into the following modules — 452 min from here to the end of the course.