← Platform Fundamentals & Architecture Review

Compute: serverless, classic, and access modes

Compute is where your money goes and where most of your failures happen. There are three decisions: which compute type, which access mode, and which runtime version. Get all three right and most performance and permission problems never appear.

The three compute types

Databricks' own docs organize compute into three families (Compute, page updated 2026-07-10):

Serverless is GA and is the default in most workspaces — no enablement step needed (Serverless compute, updated 2026-07-10). Treat it as the default choice and justify departures from it, not the reverse. That's an inversion from how the platform was taught before 2025 and it catches experienced people out.

Reach for classic when you specifically need:

• Traffic to route through your own VNet/VPC and firewall
• GPUs, or a library that needs a cluster-scoped init script
• Scala or R workloads
• Cluster policies with fine-grained instance-type control

Cost intuition

Serverless removes idle cost, which is usually the biggest source of waste — clusters sitting warm while an analyst is at lunch. But serverless bills at a higher rate per unit of work. The rule of thumb that survives contact with reality:

• Bursty, interactive, unpredictable → serverless wins (you were paying for idle).
• Long, steady, saturating jobs → classic can be cheaper (you were never idle anyway).

Don't guess. Module E9 shows how to answer this from your own system.billing.usage table rather than from a blog post.

Access modes — renamed, and it matters

This is a rename that breaks older material, so read carefully.

Sources: Dedicated access mode — the UI shows "Dedicated (formerly: Single-user)"; the standard-mode doc is titled "…standard access mode (formerly shared access mode)". Configure compute confirms Auto as the default.

The reason this shows up as an exam question and a real-world blocker: access mode determines what your code is allowed to do. Standard mode enforces isolation between users, which historically restricted certain operations — some RDD APIs, some Scala paths, certain init scripts. If a notebook works on your dedicated cluster and fails on the shared job cluster, access mode is the first thing to check, not the code.

Runtime versions

The Databricks Runtime (DBR) is the packaged image: a specific Apache Spark version, a specific Delta Lake version, Python, and a pile of libraries.

Verified from the runtime support matrix on 2026-07-30:

Practical guidance: run LTS in production. Non-LTS releases have short support windows and force upgrades on someone else's schedule. Today that means DBR 18 LTS for new work, or 17.3 LTS if you need a version that's been in the field longer.

⚠️ DBR 13.3 LTS goes out of support 2026-08-22 — within weeks of this writing. If you have jobs pinned to it, that's an action item, not a reading item.

Note also that Lakeflow pipelines use channels rather than pinned versions: CURRENT maps to DBR 17.3 and PREVIEW to DBR 18 (pipelines release notes, 2026-07-14).

Photon

Photon is a vectorized query engine written in C++ that replaces parts of Spark's JVM execution for SQL and DataFrame operations. It's not a different API — you don't write Photon code. You enable it, and qualifying operations get faster.

It helps most on scans, filters, joins, and aggregations over Delta/Parquet. It does nothing for arbitrary Python UDFs, because those can't be vectorized into its engine. If your job is one big Python UDF, Photon will not save you; rewriting the UDF might.

Primary source, if you want the engineering detail: Photon: A Fast Query Engine for Lakehouse Systems (SIGMOD 2022).

A decision checklist

Work top to bottom; stop at the first line that applies.

1. Pure SQL / BI workload? → SQL warehouse (serverless).
2. Need Scala, R, GPUs, or init scripts? → classic, dedicated access mode.
3. Traffic must traverse your own firewall? → classic in your VNet/VPC.
4. Everything else → serverless.

Then: pick the latest LTS runtime, and enable Photon unless you've measured that it doesn't help your specific workload.

Check yourself

1. What are Dedicated and Standard access modes called in a 2024 tutorial?
2. Your notebook runs fine interactively but fails when scheduled as a job. Name two compute-related causes before you look at the code.
3. Which LTS runtime would you pin a new production pipeline to today, and why not the newest GA?
4. When does Photon not help?

Teaching this section

• Open with the rename table. Half your room learned "Single user / Shared." Fixing the vocabulary in the first two minutes prevents confusion for the rest of the day.
• Live check: have everyone open their own workspace and read off the access mode and DBR version of the cluster they use daily. Someone will be on an out-of-support runtime. That's a better lesson than any slide.
• Debate prompt: "Serverless is always cheaper — true or false?" It's false, and arguing it out teaches the idle-versus-rate tradeoff better than stating it.
• Timing: 25 minutes presented, 10 for the live check.