Schema Drift in ETL: when data changes and your code does not notice

Sat, 10 Oct 2026 00:00:00 +0200

Every morning, an ETL receives a CSV, converts ages to integers, and loads customers. Yesterday it worked with this file:

id,name,age
1,John,25
2,Mary,30

Today, the provider adds a country:

id,name,age,country
1,John,25,USA
2,Mary,30,UK

A few days later, another surprise arrives:

id,name,age,country
1,John,25,USA
2,Mary,unknown,UK

An additional column may be harmless if we select fields by name and allow extra columns. If we unpack each row into three variables, it may break the load. unknown may cause an age conversion error, silently become null, or lead a subsequent inference to treat the entire column as text.

Compatibility depends on the contract and the code. Even the last case may be a violation involving individual values without the provider changing its declared schema. The question is: if we know which version of the code processed a file, can we also establish its data structure and when it stopped being compatible with our ETL?

Detecting a change does not approve it

Schema Drift describes divergence between the expected and observed structure: new columns, missing fields, or different types. In a CSV, whose values are text, observed types depend on how we interpret them.

Schema Evolution means managing those changes: agreeing on versions, deciding what changes we accept, and adapting consumers. Adding an optional field may preserve compatibility; removing a key we need will usually break it.

A Data Contract records the agreement between producer and consumer: fields, types, nullability, formats, units, quality expectations, and delivery conditions. Making it operational requires checks and a policy for violations. Recording a change, accepting it, and transforming the data are separate decisions.

For example, we might allow new columns, reject a missing id, and quarantine rows with invalid ages. That policy should be versioned: “the file could be read” tells us less than “it satisfied contract 2.1.”

Git tells only part of the story

Git can identify deployed code, but it does not automatically record what an external provider sent. Consider this sequence, where A, B, and C represent observed structures rather than already approved contracts:

Reception date	Received schema	ETL version
01 Sep	Schema A	v1.0
12 Sep	Schema B	v1.0
15 Sep	Schema B	v1.1
20 Sep	Schema C	v1.1
25 Sep	Schema C	v1.2

If v1.0 only supported A and v1.1 only supported A and B, batches received between 12 and 15 September, and between 20 and 25 September, are potentially affected. The diagram’s bands mark those review periods; they do not establish incompatibility on their own. We must consult contracts, validations, and each run’s results.

For each batch, we should record:

Executed code: a Git commit or immutable artifact, as well as a readable label.
Observed schema and applied contract: version identifiers and their descriptors.
File identity: source, batch identifier, and content hash; also the stored object version when available.
Reception and processing: separate timestamps with time zones and a run identifier.
Outcome: checks, rejections, and produced outputs.

There are also two clocks: when the source changed and when we detected it. A nightly inspection only establishes what was observed that night. Without provider events or historical files, we may be able to bound the change between two observations, but cannot assign an exact time.

A metadata history does not automatically preserve original content either. Reproducing the calculation also requires the data, dependencies, and ETL configuration, subject to the relevant retention policy.

Failures that leave the pipeline green

An amount moves from euros to cents: 19.95 becomes 1995. Both can be valid numbers. A list of columns and types will not detect the unit change; we need declared units, business rules, or a statistical comparison that flags the jump.

A date changes from 2026-10-09 to 09/10/2026. If the contract requires a format, we can reject it. If the schema merely says “text,” everything looks correct; a permissive parser may also confuse day and month for ambiguous dates.

La Coruña appears as La CoruÃ±a because bytes were interpreted incorrectly. It is still text, and may even subsequently be stored in a perfectly valid UTF-8 file. Specifying the encoding helps, but does not repair earlier corruption. This connects to encoding errors and Unicode.

These problems require combining structure, semantics, and Data Quality. An age range, an explicit currency, or a list of valid places catches issues outside schema analysis. A different distribution may also reflect a real business change: an alert calls for investigation, not automatic correction.

What each tool contributes

OpenMetadata: a shared memory for the ecosystem

OpenMetadata centralizes metadata about assets, owners, and relationships. Its version history includes schema changes and other metadata. It uses major and minor versions; deleting a column is a documented example of an incompatible change. That catalog classification does not replace the compatibility requirements of our particular consumer.

Observability alerts can monitor added, deleted, or updated columns, test results, and pipeline states. We can filter events and route them to email, chat channels, or webhooks. Its table tests check expected columns and counts, among other things; profiling supplies additional metrics.

It fits situations where several teams need a common catalog spanning databases, storage, pipelines, and analytics. It requires a platform deployment and configured ingestion and connectors. A change not yet ingested remains outside its knowledge: checking every CSV on arrival requires instrumenting that entry point.

OpenLineage: connecting data, code, and runs

OpenLineage defines an open standard for lineage. A job identifies logical work; a run, one execution; a dataset, an input or output. Facets add metadata to these entities and to a run’s inputs or outputs.

The schema facet describes fields and types. The source code location facet supports repository and version information. When our event producer supplies them, we can connect a run to its datasets and executed code. We must emit the actual reference for the deployed artifact rather than assume it matches the current branch.

Clients and integrations publish events that a backend receives and retains. OpenLineage does not inspect a CSV on its own or decide whether a change is compatible. It requires a reader or integration that supplies the schema, plus a system that compares observations, stores history, and alerts. Its value grows when we need to trace impact across several engines.

DataHub: catalog and history, with differences between editions

DataHub provides schema history for inspecting added or removed fields and type changes. This capability is available in Core, its open source edition, and Cloud. The catalog and lineage help investigate dependencies, provided we ingest those metadata.

Data Contracts group assertions about an asset. The official Core and Cloud comparison includes contracts in both editions; with Core, we can execute checks outside DataHub and publish their results, for example from dbt or Great Expectations.

Native Schema Assertions belong to DataHub Cloud Observe. They can require the schema to contain specified columns and types or to match exactly. They are evaluated when the ingested schema changes; compared types are high level. Cloud adds native monitoring and anomaly detection. Having the open assertion model does not mean having the commercial execution service.

It is an option for organizations with multiple sources and governance and discovery needs. Operating the catalog and its ingestion processes costs more than adding a library to a small ETL.

Frictionless: start with the file that just arrived

Frictionless can describe, read, and validate tabular data from Python. It can inspect a CSV, infer names and types, save a schema descriptor, and validate another delivery against it. It is useful for checking files close to reception without first deploying a catalog.

This example uses Frictionless 5.20.0, checked against the Resource, Schema, and validation documentation. Run it in a new working directory: it creates three files. The Spanish filenames are kept so both versions use the same executable example.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install frictionless==5.20.0

from pathlib import Path
from frictionless import Resource, Schema

Path("clientes.csv").write_text(
 "id,name,age,country\n1,John,25,USA\n2,Mary,30,UK\n",
 encoding="utf-8",
)
Path("clientes-cambio.csv").write_text(
 "id,name,age,country\n1,John,25,USA\n2,Mary,unknown,UK\n",
 encoding="utf-8",
)

original = Resource(path="clientes.csv", encoding="utf-8")
original.infer()
print([(field.name, field.type) for field in original.schema.fields])
original.schema.to_json("clientes.schema.json")

report = Resource(
 path="clientes-cambio.csv",
 encoding="utf-8",
 schema=Schema("clientes.schema.json"),
).validate()

print(report.valid)
print(report.flatten(["rowNumber", "fieldNumber", "type"]))

It infers id and age as integer, and name and country as string. The second file returns False and [[3, 3, 'type-error']]: its third row, including the header, contains an incompatible age.

The key detail is validating against the earlier schema. If we infer afresh for every delivery and accept the result, we may normalize the problem as a new type. Inference is a proposal rather than a contract: it depends on the sample and detector configuration. Identifiers such as 00123 should be declared as text when their meaning requires it.

Frictionless also validates constraints and supports additional checks. We would need to implement or integrate history, compatibility policy, run records, and alerts. Its readers can handle tabular JSON; nested documents require a defined representation for objects, arrays, and optional fields.

whylogs: observe content through profiles

whylogs generates compact profiles with statistical properties, missing values, and configurable metrics. We can retain profiles from different batches, compare them, and apply constraints to investigate changes in distributions, ranges, or proportions of nulls.

A profile can help show that amounts jumped in scale, for example. It does not prove they are now cents. It also records column properties that allow us to observe changes in types or presence, but a profile does not replace a contract or decide schema compatibility. Summarized or approximate statistics require interpreting the signal.

The library works locally. Historical storage, periodic comparisons, and alert delivery require an application or integration; the commercial WhyLabs service is an additional option. Profiles reduce the need to retain raw values, although that does not guarantee all metadata are harmless from a confidentiality perspective.

Compare responsibilities, not just checkboxes

In this table, “integration” means we must provide or connect that component. Complexity is an indicative assessment for this use case, not a performance measurement.

Tool	Main purpose	Schema changes	Schema history	Run traceability	Statistical analysis	Deployment	Open source
OpenMetadata	Catalog and observability	On ingested metadata; configured tests and alerts	Asset and metadata versions	Lineage and pipelines through connectors; batch detail needs instrumentation	Configurable profiling	Platform and ingestion: greater complexity	Yes; additional commercial services
OpenLineage	Lineage event standard	Carries schema; comparison in integration/backend	Depends on backend and retained events	Emitted jobs, runs, datasets, and facets	Can carry metrics; external calculation	Instrumentation plus backend	Yes
DataHub	Catalog and governance	History in Core; native Schema Assertions in Cloud Observe	Available in Core and Cloud	Lineage and job metadata depending on integration	Ingested profiles; native anomalies in Cloud	Core platform or Cloud service	Open Core; commercial Cloud
Frictionless	Describe and validate files	Inference and validation against schema; integrate historical comparison	Exportable descriptors; registry to build	Run association to build	Basic statistics and checks; no continuous statistical monitor	Library/CLI: lower complexity	Yes, MIT
whylogs	Data profiles	Column signals; comparison and contract to integrate	Storable profiles; structural versioning to integrate	Run association to integrate	Its central function	Library: lower complexity; additional monitoring	Yes; WhyLabs is an additional service

These tools complement one another. Frictionless can validate inputs; whylogs can profile them; OpenLineage can carry their relationship with a run; a catalog can retain shared context. We do not need all five to start.

Do we need an LLM?

An LLM can help interpret a change or propose a transformation. Many basic checks can be handled with type inference, rules, regular expressions, and statistical analysis, producing results that are easy to review.

Adding one introduces infrastructure cost, latency, and outputs that are not necessarily deterministic. It also requires deciding which data or metadata it may receive and validating proposed transformations. Local models exist: avoiding an external service changes infrastructure requirements but does not remove these issues. A suggested conversion should never automatically become a new business rule.

A lightweight architecture we could build

A Python library independent of the ETL engine could inspect CSV/JSON and maintain historical metadata without executing transformations. This is a conceptual proposal:

CSV / JSON → Inspection → Observed schema + validation
 ↓
 Version registry
 ↓
 Batch + hash + run + commit + timestamps
 ↓
 Unknown change → Alert → Review

First, we would normalize the descriptor and generate a structural fingerprint. We would need to decide whether column order matters, how nested types are represented, and which properties enter the fingerprint. We would also record the algorithm version and inference options so a detector change is not confused with a data change. A schema fingerprint and a file hash serve different purposes.

Next, we would record immutable observations: the same schema across several batches does not mean the same content. The approved contract would have its own identifier. An unknown structure would trigger a review; a known one would still need row validation and semantic checks.

Associations between batches, runs, and code would let us query which executions combined Schema C with v1.1. Those batches would be review candidates, rather than automatic evidence of incorrect results. An explicit policy would decide whether to stop loading, isolate rows, or continue with an alert. SQLite could serve as a local starting point; process coordination and retention would require further decisions.

Reconstructing what happened

Versioning code is necessary, but insufficient to explain a pipeline. We also need to know the received data, their structure, the applied contract, and each run’s outcome. Catalogs, lineage events, and validation and profiling libraries cover different parts of that history.

If a file changes tomorrow without warning, could you identify which batches were processed with each schema and ETL version, and justify which ones need review?

Article credits

The idea and editorial supervision of this article belong to José Pérez. AI assisted its preparation, with the contributions detailed in these credits.

Metadata	Value
Editing	José Pérez
Research	AI assisted
Writing	OpenAI Codex
Model	GPT-6
Generation time	Not recorded
Human review	Pending
Sources verified	Yes

Sources were verified with AI assistance by checking the technical claims against official documentation. Human review of the final text is awaiting the author’s confirmation.

OpenMetadata on josiete.com