Detecting Silent Schema Drift Between Releases

A consumer’s pipeline broke on the new release. Every gate passed, the geometry is valid, the statistics are within tolerance, and a column that used to be an integer is now a float — or a categorical field grew a value nobody declared.

Part of Artifact Versioning & Lineage: this page is the class of change that is invisible to every quality gate and visible immediately to every consumer.

Root Cause: Quality Gates Validate Values, Not Shapes

The gates a synthetic spatial pipeline runs — geometry validity, envelope containment, statistical parity, privacy accounting — all take the schema as given and check the values inside it. None of them compares this release’s shape to the last one’s, because none of them has the last one.

That leaves a category of change that passes everything and breaks consumers on contact:

  • A column appears or disappears. Additive changes break consumers doing strict column checks; removals break everyone.
  • A dtype widens or narrows. int32 becomes int64 because a value overflowed, or a nullable integer becomes a float because a null appeared. Both are silent, both change the on-disk representation, and both break a consumer with a fixed reader schema.
  • A categorical gains a value. The generator learned a new land-use class, or a fallback introduced an unknown. A consumer with an enum mapping now has an unmapped value, and depending on their code that is either an exception or a silently dropped row.
  • Nullability changes. A column that was never null now is, because a fallback rule started firing. Nothing about the values is wrong; every consumer that assumed non-null is.
  • Units or CRS change inside an unchanged column name. The nastiest of the set, because the shape is identical and only the meaning moved.
Five schema-drift classes against the four checks that might catch each Rows are drift classes: a column appearing or disappearing, a dtype widening or narrowing, a categorical gaining a value, a column becoming nullable, and units or CRS changing behind an unchanged column name. Columns are checks. Geometry validity passes all five, because every geometry involved is valid. Statistical parity passes all five, because none of them moves a distribution outside tolerance — a widened integer holds the same values, and a new categorical value is a small share of rows. A schema-shape comparison against the previous release catches the first four: it sees the column set change, the dtype change, the category set change and the nullability change. It does not catch the fifth, because a float column of distances is byte-identical in metres and in feet. Only a contract that declares the unit and the CRS per column catches that one, and it catches it by comparing declarations rather than data. The conclusion is stated underneath: a pipeline with excellent quality gates and no shape comparison has zero coverage of this entire class, and the coverage it needs costs one stored description per release. The quality gates catch none of these Drift class geometry statistics shape diff contract units/CRS column added or removed passes passes catches passes dtype widened or narrowed passes passes catches passes categorical gained a value passes passes catches passes column became nullable passes passes catches passes unit or CRS changed silently passes passes passes catches The last row is why a contract carries units and a CRS, not only names and types. A float column of distances is byte-identical in metres and in feet — the change is undetectable from the data, and detectable in one line from the declaration.
Five drift classes against the gates that might catch them: the quality gates see none, and a shape comparison sees four of the five.

The fifth class is the one that stays invisible even to a schema check, and it is the reason a contract carries units and a CRS rather than only names and types.

Minimal Reproducer: Two Releases That Both Pass

python
import pandas as pd

previous = pd.DataFrame({
    "cell_id": pd.array([1, 2, 3], dtype="int32"),
    "land_use": pd.Categorical(["residential", "retail", "residential"]),
    "density": [12.4, 88.1, 9.0],
})

current = pd.DataFrame({
    "cell_id": pd.array([1, 2, 3], dtype="int64"),          # widened
    "land_use": pd.Categorical(["residential", "retail", "unknown"]),   # new value
    "density": [12.4, 88.1, None],                          # now nullable
})

for name, df in (("previous", previous), ("current", current)):
    print(name, {c: str(df[c].dtype) for c in df.columns})

Every value in current is legitimate. The geometry gate has nothing to look at, the statistical gate sees a mean within tolerance, and the privacy ledger is unaffected. A consumer reading with a fixed Arrow schema fails on the first row.

Fix: Compare Shapes at Promotion, With a Compatibility Policy

1 — Extract a canonical schema description

python
from dataclasses import dataclass, asdict


@dataclass(frozen=True)
class ColumnShape:
    name: str
    dtype: str
    nullable: bool
    categories: tuple | None      # for categoricals, the declared value set
    unit: str | None              # from the contract, not inferred
    crs: str | None               # for geometry columns


def describe(df, contract: dict) -> list[ColumnShape]:
    out = []
    for col in sorted(df.columns):
        cats = tuple(sorted(map(str, df[col].cat.categories))) \
            if str(df[col].dtype) == "category" else None
        out.append(ColumnShape(
            name=col,
            dtype=str(df[col].dtype),
            nullable=bool(df[col].isna().any()),
            categories=cats,
            unit=contract["columns"].get(col, {}).get("unit"),
            crs=contract["columns"].get(col, {}).get("crs"),
        ))
    return out

Sorting the columns matters for the same reason it does everywhere else in this area: an unsorted description changes when nothing changed, and a description that changes spuriously is one nobody reads.

Note that unit and crs come from the contract rather than from the data. They cannot be inferred — a float column of distances looks identical in metres and feet — which is exactly why the fifth drift class is invisible without them.

2 — Classify each difference by compatibility

python
BREAKING, ADDITIVE, SAFE = "breaking", "additive", "safe"


def classify(prev: ColumnShape | None, cur: ColumnShape | None) -> list[tuple[str, str]]:
    if prev and not cur:
        return [(BREAKING, f"column {prev.name} removed")]
    if cur and not prev:
        return [(ADDITIVE, f"column {cur.name} added")]
    issues = []
    if prev.dtype != cur.dtype:
        widening = (prev.dtype, cur.dtype) in {("int32", "int64"), ("float32", "float64")}
        issues.append((ADDITIVE if widening else BREAKING,
                       f"{cur.name}: dtype {prev.dtype} → {cur.dtype}"))
    if cur.nullable and not prev.nullable:
        issues.append((BREAKING, f"{cur.name}: became nullable"))
    if prev.categories and cur.categories:
        added = set(cur.categories) - set(prev.categories)
        removed = set(prev.categories) - set(cur.categories)
        if added:
            issues.append((ADDITIVE, f"{cur.name}: new categories {sorted(added)}"))
        if removed:
            issues.append((BREAKING, f"{cur.name}: categories removed {sorted(removed)}"))
    if prev.unit != cur.unit:
        issues.append((BREAKING, f"{cur.name}: unit {prev.unit} → {cur.unit}"))
    if prev.crs != cur.crs:
        issues.append((BREAKING, f"{cur.name}: CRS {prev.crs} → {cur.crs}"))
    return issues or [(SAFE, f"{cur.name}: unchanged")]

The classification is the policy, and writing it down is most of the value. “Widening an integer is additive, becoming nullable is breaking” is a decision somebody has to make, and having it in code means it is made once rather than argued each time.

3 — Bind the classification to the version bump

python
def required_bump(issues: list[tuple[str, str]]) -> str:
    kinds = {kind for kind, _ in issues}
    return "major" if BREAKING in kinds else ("minor" if ADDITIVE in kinds else "patch")


def gate_promotion(prev_schema, cur_schema, declared_version: str, previous_version: str):
    issues = diff_schemas(prev_schema, cur_schema)
    need = required_bump(issues)
    got = bump_kind(previous_version, declared_version)
    if got != need:
        raise SystemExit(
            f"schema requires a {need} bump, release declares {got}\n" +
            "\n".join(f"  {k}: {m}" for k, m in issues if k != SAFE)
        )

This is what makes the version number mean something. Without it, the semantic version is a label somebody types; with it, it is derived from the schema difference and a consumer reading “major” knows exactly what class of change to expect.

Schema differences mapped to the version bump each requires Nine differences are listed with the bump each forces. Breaking, requiring a major bump: a column removed, because every consumer reading it fails; a dtype narrowed, because values may no longer fit; a column becoming nullable, because a consumer who assumed non-null is now wrong; a categorical value removed, because a mapping loses a key; and a unit or CRS change, because the meaning moved while the shape did not. Additive, requiring a minor bump: a column added, which breaks only consumers doing strict column checks; a dtype widened, which is representationally safe; and a categorical value added, which breaks only consumers with a closed mapping. Safe, leaving a patch: no shape difference at all, meaning only values changed. A footer states what binding the bump to the diff buys: the version stops being a label somebody types and becomes a derived fact, so a consumer reading major knows exactly which class of change to expect and can decide whether to upgrade without reading a changelog. Derive the bump from the diff — then the version means something BREAKING → major column removed every consumer reading it fails dtype narrowed values may no longer fit became nullable a consumer who assumed non-null is now wrong categorical value removed a mapping loses a key unit or CRS changed the meaning moved, the shape did not ADDITIVE → minor column added breaks only strict column checks dtype widened representationally safe categorical value added breaks only a closed mapping SAFE → patch no shape difference only values changed A derived version is a version a consumer can act on. Reading "major" tells them which class of change to expect, without reading a changelog or asking anybody.
The compatibility policy as a table: every difference maps to a bump, and the bump is derived rather than declared.

Verification Step: Run the Comparison Against the Registry

python
def test_schema_is_compatible_with_the_current_release(candidate, registry, contract):
    current = registry.current(candidate["name"])
    prev = load_schema(current["artifact_sha"])
    cur = describe(candidate["frame"], contract)
    issues = [i for i in diff_schemas(prev, cur) if i[0] != SAFE]
    assert required_bump(issues) == bump_kind(current["version"], candidate["version"]), issues


def test_contract_declares_units_for_every_numeric_column(contract, frame):
    missing = [c for c in frame.columns
               if pd.api.types.is_numeric_dtype(frame[c])
               and not contract["columns"].get(c, {}).get("unit")]
    assert not missing, f"no unit declared for {missing} — unit drift will be invisible"

The second test is the one that closes the fifth class. A numeric column with no declared unit is a column whose meaning can change without any detectable difference, and requiring the declaration is the only defence.

Edge Cases & Gotchas

The first release has nothing to compare against. Record its schema and let the check pass with a note rather than skipping it silently, so the absence is visible in the promotion record rather than looking like a pass.

Three homes for a schema description, by availability, fidelity and timing Three rows. Inferring the description from the file at read time needs nothing stored, but the previous release's description is unavailable unless the previous file is also opened; units and CRS cannot be inferred at all, because they are not properties of the bytes; and the comparison can only run after both files exist, which is after promotion. Storing the description in the file's own metadata makes it travel with the data, which is genuinely useful, but the previous release's metadata still requires opening the previous file; units and CRS survive only if the writer put them there; and the comparison still runs late. Storing it in the registry beside the release makes the previous description a single lookup, preserves units and CRS because they come from the contract rather than the data, and lets the comparison run before promotion — which is the only point at which failing it costs nothing. A footer notes that the three are not exclusive: writing the description into the file as well is worth doing for consumers, and it is the registry copy that the promotion gate reads. Only one of the three lets the comparison run before promotion Where it lives previous available? units and CRS survive? can gate promotion? inferred at read time only by opening the old file ✗ ✗ in the file's metadata only by opening the old file if the writer wrote them ✗ in the registry one lookup ✓ ✓ The three are not exclusive. Write the description into the file too — it helps consumers. The promotion gate reads the registry copy, because that is the one available before the release exists. Failing a compatibility check after promotion is a withdrawal; failing it before is a version bump. The difference is entirely a matter of when the previous description can be read.
Only a description stored in the registry is available before the release exists — which is the only point at which failing the check costs nothing.

A column is renamed. The diff sees a removal and an addition, and reports a breaking change plus an additive one, which is correct but unhelpful. A rename map in the contract lets the check report it as a single rename and still classify it as breaking.

Categories that legitimately grow every release. Some vocabularies are open by nature. Declare the column as open in the contract, and check that new values match a declared pattern rather than a declared set — an unbounded enum is still checkable, just differently.

Nullability that flickers. A column is nullable in one release and not the next because the fallback happened not to fire. Derive nullability from the contract’s declaration rather than from the data, and check the data against it; otherwise the diff reports a breaking change every time the input happens to be complete.

Making the Check Cheap Enough to Always Run

A compatibility check that runs sometimes is a compatibility check that will be missing when it matters, so it is worth making it fast enough to be unconditional.

The schema description is small — a few hundred bytes for a wide table — so storing one per release costs nothing and the comparison is a dictionary diff. What costs is producing it, if the implementation reads the whole artifact to find out whether a column is nullable or which categories are present. On a large release that is a full scan.

Two things avoid it. Most columnar formats carry per-column statistics in their footer — null counts, min and max, and for dictionary-encoded columns the dictionary itself — so the description can be assembled from metadata without touching a data page. And where a statistic genuinely requires the data, compute it during generation rather than afterwards: the generator already visits every value, and accumulating a null count and a category set as it goes is free relative to what it is already doing.

The result is a check that adds milliseconds to promotion rather than minutes, which is the difference between a gate that runs on every release and one somebody disables during a busy week and forgets to re-enable.

There is a second reason to compute the description during generation rather than from the artifact: it is then a declaration rather than an observation. A description derived from the data says what happened to be there this time; one produced by the generator from the contract says what was supposed to be there, and the difference between the two is itself worth asserting.