← Data Ingestion and Loading

▶ Listen · 7 min

Choosing your ingestion layer

The three layers

Databricks organizes ingestion into three layers, ordered from most managed to least (Ingest data, 2026-07-10):

1. Managed connectors (Lakeflow Connect) — handles "source-specific authentication, CDC, edge case handling, long-term API maintenance, automated retries, automated schema evolution."

Read that list again. Every item is something you would otherwise build, own, and be paged about at 3am. When Salesforce changes an API version, somebody has to notice and adapt. With a managed connector, that somebody is Databricks.

2. Lakeflow Pipelines — "manage orchestration, monitoring, data quality, errors, and more. They offer more automation and less overhead than Structured Streaming."

3. Structured Streaming — "end-to-end fault tolerance with exactly-once processing guarantees using Spark APIs." Maximum control, maximum responsibility.

The direction of travel

"Databricks recommends starting with the most managed layer. If it doesn't satisfy your requirements (for example, if it doesn't support your data source), drop down to the next layer."

This is worth internalizing because it runs against engineering instinct. The interesting code lives at layer 3. Given a blank notebook, most engineers start writing spark.readStream.

The professional move is the opposite: start at layer 1, and only descend when you hit a genuine wall. Each step down is a step you now own forever.

Expect an exam question on this. The answer is always "start managed, drop down."

Where data lands: volumes, external locations, mounts

Before ingesting, you need somewhere for files to land — and this is where a lot of stale material will lead you astray.

Volumes are "Unity Catalog objects that enable governance over non-tabular datasets" (Volumes, 2026-06-25). Managed volumes use UC-controlled storage and need no credential management; external volumes add governance over existing cloud paths without moving the data.

DBFS mounts are unambiguous in current docs (Mounts, 2026-06-17):

"DBFS mounts are a deprecated pattern and are not compatible with Databricks serverless compute architecture." "Mounted data does not work with Unity Catalog."

Combined with E0's finding that new accounts are provisioned without DBFS access at all: any tutorial building an ingestion pipeline on /mnt/ paths is teaching you something you cannot deploy.

Volumes aren't just tidier — they're faster

This is the detail that separates people who've read the docs from people who've run this in production. From the file notification docs:

"Use Unity Catalog volumes: Create a separate external volume for each path or subdirectory that Auto Loader loads data from. Supply volume paths (for example, /Volumes/catalog/schema/volume) to Auto Loader instead of cloud storage URLs (for example, s3://bucket/path). This improves file discovery performance because the file events service can scope file discovery to only the relevant objects, rather than iterating over all objects in the external location."

Skip this and you can hit a real, documented error:

com.databricks.sql.util.UnexpectedHttpStatus: Too many requests. Please wait a moment and try again.

Cause: multiple streams reading different subpaths under a single external location, without volumes to scope discovery. Each stream iterates everything, and you rate-limit yourself against your own storage account.

Rule: one external volume per ingestion path. Not one volume for the whole bucket.

A decision framework

Is the source a SaaS app, or a database needing CDC?
├─ YES → Is there a Lakeflow Connect managed connector?
│        ├─ YES → USE IT. Stop here.
│        └─ NO  → drop to Lakeflow Pipelines
└─ NO (files in cloud storage, or a message bus)
         → Lakeflow Pipelines + Auto Loader
              └─ need custom state / arbitrary sinks / exotic logic?
                    → Structured Streaming

Two clarifications people get wrong:

Auto Loader isn't a fourth layer. It's the file-ingestion mechanism (cloudFiles), usable from layer 2 or layer 3. "Auto Loader vs Lakeflow Pipelines" is a category error — you typically use Auto Loader inside a pipeline.

Managed connectors don't cover files in your own storage. Lakeflow Connect's file-source connectors handle external services like SharePoint and Google Drive. Files already landed in your own ADLS/S3/GCS are an Auto Loader job.

Check yourself

1. Your source is Salesforce. Where do you start, and why not Structured Streaming?
2. Why is one volume per ingestion path better than one volume per bucket?
3. A 2023 tutorial mounts storage to /mnt/raw and runs Auto Loader against it. Two reasons that's wrong today.
4. Is Auto Loader a layer in the three-layer model?

Answers

1. Layer 1 — check for a Lakeflow Connect managed Salesforce connector. Databricks explicitly recommends starting most-managed. Structured Streaming would mean owning Salesforce auth, API pagination, CDC, schema drift, and retries yourself — forever.
2. It lets the file events service scope discovery to relevant objects instead of iterating the whole external location. Without it, multiple streams over one location can trigger UnexpectedHttpStatus: Too many requests.
3. (a) DBFS mounts are deprecated, incompatible with serverless, and don't work with Unity Catalog. (b) New accounts have no DBFS access at all, so it can't run. Use a volume.
4. No. It's the file-ingestion mechanism (cloudFiles), used within Lakeflow Pipelines or Structured Streaming.

Teaching this section

• Lead with the counter-intuitive bit — engineers want to write streaming code. State plainly that the recommended direction is top-down, and that every layer you descend is one you own forever.
• Good discussion: "What do you actually lose by using a managed connector?" Real answers exist (less control, connector coverage gaps, less visibility). It's a better conversation than presenting managed as strictly superior.
• Watch for: the "Auto Loader vs pipelines" category error. Correct it the first time you hear it — it confuses everything downstream.
• Timing: 20 min presented, 10 discussion.