← Data Ingestion and Loading

▶ Listen · 11 min

Auto Loader: schema, evolution, rescue

Auto Loader is a Structured Streaming source, invoked with format("cloudFiles"), that incrementally processes new files as they arrive — without you tracking what's been seen.

The minimal working example

(spark.readStream.format("cloudFiles")
  .option("cloudFiles.format", "parquet")
  # The schema location directory keeps track of your data schema over time
  .option("cloudFiles.schemaLocation", "<path-to-schema>")
  .load("<path-to-source-data>")
  .writeStream
  .option("checkpointLocation", "<path-to-checkpoint>")
  .start("<path-to-target>"))

Source: Schema inference and evolution (2026-07-10)

Two locations, two jobs — don't confuse them:

"You can use the same directory as your checkpointLocation. If using Lakeflow pipelines, Databricks manages schema location automatically."

That last clause matters: inside a Lakeflow pipeline you don't set schemaLocation at all. A lot of confusion comes from copying standalone-streaming examples into pipeline code.

The options that matter

Verified from the Spark API options reference (2026-07-27):

inferColumnTypes defaults to false — and the consequence is stated plainly in the docs: "By default, columns are inferred as strings when inferring JSON datasets." So your beautifully typed JSON arrives as all strings unless you ask otherwise. This surprises people constantly.

Schema evolution modes

This is the highest-value table in the module.

Source: Schema inference and evolution (2026-07-10)

The default failing is a feature

Newcomers see addNewColumns throw UnknownFieldException and file a bug. It isn't one.

The sequence: Auto Loader spots a new column → infers the schema of the latest batch → writes the updated schema to the schema location → then throws. So on restart, the schema is already correct and processing resumes.

The design assumes an orchestrator that restarts the stream. Databricks recommends running Auto Loader under Lakeflow Jobs with automatic restart. Configure that, and schema evolution is invisible — one failed run, one automatic restart, business as usual.

⚠️ A real documentation inconsistency

The two relevant pages disagree, as of 2026-07-30:

• The API options reference lists four values: addNewColumns, none, rescue, failOnNewColumns.
• The schema page documents five, adding addNewColumnsWithTypeWidening.

Both were fetched the same day. Learn the five-value list — the schema page is the behavioral reference — but know the options table omits type widening. Test it in your workspace before relying on it in production.

The rescued data column

Your insurance policy against silent data loss:

"When Auto Loader infers the schema, Auto Loader automatically adds a rescued data column to your schema as _rescued_data. […] The rescued data column ensures that Auto Loader rescues columns that don't match the schema instead of dropping them. The rescued data column contains any data that isn't parsed for the following reasons:

• The column is missing from the schema.
• Type mismatches.
• Case mismatches.

The rescued data column contains a JSON blob with the rescued columns and the source file path of the record."

Three things worth noticing.

Case mismatches count. customerID vs customerId — one becomes rescued data. That's a common real-world cause of "the column is empty and I don't know why."

It records the source file path. So when you find something in _rescued_data, you can trace it to the exact file. Invaluable during an incident.

It's automatic during inference, but not when you supply a schema. If you .schema(...) explicitly, you must set the option yourself:

df = spark.readStream.format("cloudFiles") \
  .option("cloudFiles.format", "csv") \
  .option("rescuedDataColumn", "_rescued_data") \
  .schema(<schema>) \
  .load(<path>)

⚠️ The prefix inconsistency

The behavioral page and the working code sample use rescuedDataColumn (no prefix). The API options table lists cloudFiles.rescuedDataColumn. Both appear in current docs and I found no page reconciling them.

Best current understanding: it's a reader/format option — it also applies to plain JSON/CSV readers — which is why it appears unprefixed in code, while the cloudFiles table lists a prefixed form. Use the unprefixed form in the documented sample position, and verify in your workspace.

The silent-loss combination

Here is the configuration that loses data without telling you:

# ⚠️ DANGER
.schema(my_schema)                                  # → schemaEvolutionMode defaults to "none"
# and no rescuedDataColumn set

Supply a schema and the evolution default flips to none. Under none, new columns are ignored and not rescued unless you set rescuedDataColumn. A new field appears upstream, and it silently vanishes. No error. No warning.

If you supply a schema, always set rescuedDataColumn. This is the single most valuable operational habit in the module.

Schema hints

Hints override inferred types — they're an input to inference, not a replacement for it.

.option("cloudFiles.schemaHints", "tags map<string,string>, version int")

Nested and complex types work:

.option("cloudFiles.schemaHints",
        "date DATE, user_info.dob DATE, purchase_options MAP<STRING,STRING>, time TIMESTAMP")

.option("cloudFiles.schemaHints",
        "products ARRAY<INT>, locations.element STRING, users.element.id INT, "
        "ids MAP<STRING,INT>, names.key INT, prices.value INT")

Two rules, verbatim:

"Auto Loader uses schema hints only if you do not provide a schema."

"If a column is not present at the start of the stream, you can also use schema hints to add that column to the inferred schema."

That second one is genuinely useful: a column that only appears in later files can be declared up front so downstream code doesn't break.

Hints work regardless of cloudFiles.inferColumnTypes.

Hints vs evolution — the distinction

Working patterns

Type inference on:

spark.readStream.format("cloudFiles") \
  .option("cloudFiles.format", "json") \
  .option("cloudFiles.schemaLocation", "<path-to-checkpoint>") \
  .option("cloudFiles.inferColumnTypes", "true") \
  .load("/Volumes/catalog_name/schema_name/volume_name/nested_json")

Rescue mode with an enforced schema — never fails on schema change:

spark.readStream.format("cloudFiles") \
  .schema(expected_schema) \
  .option("cloudFiles.format", "json") \
  .option("cloudFiles.schemaEvolutionMode", "rescue") \
  .load("/Volumes/catalog_name/schema_name/volume_name/source_data") \
  .writeStream \
  .option("checkpointLocation", "<path-to-checkpoint>") \
  .start("<path_to_target>")

Easy ETL with merge-on-write:

spark.readStream.format("cloudFiles") \
  .option("cloudFiles.format", "json") \
  .option("cloudFiles.schemaLocation", "<path-to-schema-location>") \
  .load("/Volumes/catalog_name/schema_name/volume_name/source_data") \
  .writeStream \
  .option("mergeSchema", "true") \
  .option("checkpointLocation", "<path-to-checkpoint>") \
  .start("<path_to_target>")

All from Common data loading patterns (2026-07-10).

Check yourself

1. What are schemaLocation and checkpointLocation each for? Can they share a directory?
2. You ingest JSON without setting inferColumnTypes. What types do you get?
3. addNewColumns throws UnknownFieldException. Bug or design? What must exist for it to work?
4. Name the config that silently drops new columns.
5. You supply .schema(...) and cloudFiles.schemaHints. What happens to the hints?
6. Which mode never fails on schema change?

Answers

1. schemaLocation stores the inferred schema and its evolution history; checkpointLocation stores stream progress. Yes, the docs explicitly allow the same directory. Inside a Lakeflow pipeline, Databricks manages schema location for you.
2. Strings — "columns are inferred as strings when inferring JSON datasets" because inferColumnTypes defaults to false.
3. Design. Auto Loader writes the updated schema before throwing, so a restart resumes correctly. It needs an orchestrator that restarts the stream — Databricks recommends Lakeflow Jobs with automatic restart.
4. Supplying an explicit .schema() (which flips the evolution default to none) without setting rescuedDataColumn. New columns are ignored and not rescued. No error is raised.
5. Ignored — "Auto Loader uses schema hints only if you do not provide a schema."
6. rescue. It never evolves the schema and never fails on schema change; everything unexpected lands in the rescued data column.

Teaching this section

• The silent-loss combination is the moment of the lesson. Demo it: enforce a schema, drop a file with an extra column, show the column vanishing with no error. Then add rescuedDataColumn and show it caught. People remember data disappearing.
• Frame UnknownFieldException as designed behavior before anyone hits it, or you'll spend the rest of the day fielding "is this broken?"
• Be honest about the doc inconsistencies. Modeling "the docs disagree, so I tested it" is more valuable than any single fact here.
• Timing: 35 min presented, 20 for the rescued-data demo.