← Data Ingestion and Loading

▶ Listen · 12 min

File detection modes across three clouds

How does Auto Loader know a new file arrived? Two mechanisms, and the newer one has two flavors. This is where the three clouds genuinely differ, so it's the most architecture-flavored lesson in E1.

Directory listing vs file notification

Directory listing (default) — "identifies new files by listing the input directory." Needs only read access to storage; no extra cloud permissions. Since DBR 9.1, Auto Loader "can automatically detect whether files are arriving with lexical ordering to your cloud storage and significantly reduce the amount of API calls needed."

File notification — "Auto Loader automatically sets up a notification service and queue service that subscribes to file events from the input directory. You can use file notifications to scale Auto Loader to ingest millions of files an hour. When compared to directory listing mode, file notification mode is faster and more scalable."

The current recommendation, verbatim:

"Databricks recommends file notification mode using file events instead of directory listing mode for most workloads."

Note this is a change from older guidance, which treated directory listing as the sane default and notification as a scale-out escape hatch. Older training material will tell you the opposite.

Two more facts worth memorizing:

"You can switch file discovery modes across stream restarts and still obtain exactly-once data processing guarantees."

"Auto Loader does not guarantee the order in which files are discovered or processed, regardless of file detection mode."

The first is reassuring — mode is not a one-way door. The second kills a whole class of bad designs: never build logic that assumes files are processed in name or timestamp order.

Source: File detection modes (2026-06-16)

Two flavors of file notification

This distinction is newer than most exam-prep material, and it's the modern answer.

File events (recommended)

"You use a single file notification queue for all streams that process files from a given external location."

Databricks sets up the subscription and queue in your storage account, and Auto Loader needs no elevated credentials of its own. You enable file events on the Unity Catalog external location, then set:

.option("cloudFiles.useManagedFileEvents", "true")

"File events provide notifications-level performance in file discovery, because Auto Loader can discover new files after the last run. Unlike directory listing, this process does not need to list all files in the directory."

Requires DBR 14.3 LTS+.

One queue for all streams is the operational win. Classic mode creates a queue per stream; run forty streams and you're managing forty queues.

Classic file notification

"You manage file notification queues for each Auto Loader stream separately."

Driven by cloudFiles.useNotifications = true. Requires elevated cloud permissions so Auto Loader can create the topic and queue itself.

Fallbacks and expiry — the gotcha

Even with file events on, Auto Loader falls back to directory listing when:

1. It's the initial load with includeExistingFiles = true, and
2. the recent-files cache expires.

"If Auto Loader runs infrequently, this cache can expire… To avoid this scenario, invoke Auto Loader at least once every seven days."

Run your stream at least weekly. A monthly pipeline on file events silently degrades to a full directory listing every run.

Options ignored under file events

useIncremental, useNotifications, cloudFiles.fetchParallelism, cloudFiles.backfillInterval, cloudFiles.pathRewrites, resourceTags.

Note backfillInterval is in that list — you can't use the backfill safety net with file events.

Runtime support matrix

Two sharp exam facts hide in there:

• UC volumes do not support classic file notification. Volumes + notifications = file events, or nothing. Given lesson 1 told you to use volumes, this effectively forces file events.
• Azure Blob Storage doesn't support file events — and recall from E0 that Blob Storage isn't compatible with Unity Catalog either. Use ADLS Gen2.

The three clouds

The architectural observation worth having ready in an interview: AWS and Azure each split the role across two services — one publishes events, another queues them. GCP uses Pub/Sub for both. Same logical pattern, three different decompositions.

Azure (your primary)

Auto Loader creates Event Grid + Queue Storage resources. Authenticate either with a Databricks service credential (DBR 16.1+, using a managed identity and an Azure Databricks access connector) or an Entra ID service principal with client ID and secret.

Required built-in roles:

• Contributor and Storage Queue Data Contributor on the storage account
• EventGrid EventSubscription Contributor on the resource group

Event semantics: "Auto Loader listens for the FlushWithClose event for processing a file." It also supports RenameFile (needs an extra API call to get size) and, from DBR 9.0+, RenameDirectory (needs a listing of the renamed directory).

Two Azure-only traps:

"Auto Loader doesn't support file notification mode for Azure premium storage accounts because premium accounts don't support queue storage."

And if Event Grid isn't registered as a resource provider on the subscription:

java.lang.RuntimeException: Failed to create event grid subscription.

Fix: register the Microsoft.EventGrid provider.

AWS

Auto Loader creates an SNS topic and an SQS queue, both prefixed databricks-auto-ingest, limited to 100 per bucket. You attach an IAM policy with statements DatabricksAutoLoaderSetup / DatabricksAutoLoaderList / DatabricksAutoLoaderTeardown, granting s3:GetBucketNotification, s3:PutBucketNotification, and the relevant sns:* / sqs:* actions.

Elevated setup permissions are needed only on the first run. Afterwards you can downgrade to a reduced DatabricksAutoLoaderUse policy — but then you "can't start new streaming queries, recreate resources after failures such as an accidentally deleted SQS queue, or use the cloud resource management API."

Event semantics: S3 emits ObjectCreated "regardless of whether it was uploaded by a put or multi-part upload."

GCP

Pub/Sub plays both roles. Setup needs list and get on the bucket and its objects, the Pub/Sub Publisher role on the GCS service account (found under Cloud Storage → Settings → Cloud Storage Service Account), and a role carrying pubsub.subscriptions.* and pubsub.topics.* permissions.

Event semantics: GCS emits OBJECT_FINALIZE "when a file is uploaded, which includes overwrites and file copies. Failed uploads do not generate this event."

Two operational warnings

Auto Loader doesn't clean up after itself:

"Auto Loader doesn't automatically tear down file notification resources."

Delete a pipeline and its queue and topic remain, quietly costing money and counting against your per-bucket limit. Put teardown in your decommissioning runbook.

Cloud events aren't perfectly reliable:

"Cloud providers do not guarantee 100% delivery of all file events… Databricks recommends that you trigger regular backfills with Auto Loader by using the cloudFiles.backfillInterval option… Triggering regular backfills does not cause duplicates."

So notification mode alone can miss a file. The mitigation is a periodic backfill — and because Auto Loader tracks what it has processed, backfilling is safe.

⚠️ But remember backfillInterval is ignored under file events. Under file events your safety net is instead the requirement to run at least every seven days.

Check yourself

1. Which detection mode do the docs currently recommend for most workloads?
2. Can you switch modes on an existing stream without breaking exactly-once?
3. Why can't you use classic file notification with a UC volume?
4. What breaks if a file-events pipeline runs monthly?
5. What's structurally different about GCP's notification setup?
6. Two Azure-specific reasons file notification might fail outright?
7. You delete a pipeline. What's left behind?

Answers

1. File notification mode using file events — a reversal of older guidance.
2. Yes: "You can switch file discovery modes across stream restarts and still obtain exactly-once data processing guarantees."
3. The support matrix lists classic file notification as Unsupported on UC volumes. Volumes get directory listing (13.3 LTS+) or file events (14.3 LTS+).
4. The recent-files cache expires, so it silently falls back to full directory listing. Run at least every seven days.
5. GCP uses Pub/Sub for both the subscription and queue roles; AWS and Azure each split those across two services (SNS+SQS, Event Grid+Queue Storage).
6. (a) Premium storage accounts don't support queue storage, so notification mode is unsupported. (b) The Microsoft.EventGrid resource provider isn't registered → "Failed to create event grid subscription."
7. The notification queue and topic — Auto Loader doesn't tear them down. They cost money and consume the per-bucket/account limit.

Teaching this section

• Draw the three clouds side by side and let the group spot that GCP collapses two boxes into one. Discovering the asymmetry beats being told it.
• The teardown warning lands best as a war story: an account at its 100-queue limit because nobody cleaned up dead pipelines. Ask who owns decommissioning on their team. Usually nobody.
• For a multi-cloud audience, assign each cloud to a group and have them present the permission model. Fifteen minutes, and it surfaces that the pattern is identical while the primitives differ — which is the actual architectural lesson.
• Timing: 30 min presented, 15 for the cloud comparison exercise.