Auditing a Run That Cannot Be Reproduced

You have followed the restoration procedure exactly, the artifact hash does not match, and there is no error anywhere — two files that should be identical simply are not.

Part of Seed Management & Run Reproducibility: where reproducing a release from its manifest covers the happy path, this page is what to do when it fails. The method is bisection, and the reason it works is that a divergence has a location — in space, in the pipeline, and in the environment — and each of those can be narrowed independently.

Root Cause: A Hash Mismatch Is Not a Diagnosis

An artifact hash is a single bit of information: same or different. Everything useful comes from asking the same question at a finer granularity, and there are three axes to narrow along.

  • Spatially, by comparing per-partition hashes. If one tile differs and the rest match, the cause is inside that tile’s generation and every stage that is global has been eliminated.
  • By stage, by hashing the intermediate output of each stage rather than only the final artifact. If the sampler’s output matches and the attribute stage’s does not, four fifths of the pipeline is eliminated.
  • By environment, by re-running the same stage under systematically varied conditions — one worker instead of eight, a different thread count, a different machine — and observing which variation reproduces the divergence.

The order matters, because each axis is cheaper than the one after it and eliminates more. Spatial bisection costs one comparison per partition and can usually be done on artifacts you already have. Stage isolation costs a re-run. Environment variation costs several.

Three narrowing axes for a failed reproduction, by cost and eliminating power Three bands in the order they should be attempted. Spatial narrowing compares per-partition digests that already exist alongside the artifact; it costs zero re-runs, and its output alone distinguishes four cases — one partition differing, all of them differing, a contiguous block differing, or every partition matching while the merged artifact does not, which localises the fault to the merge before anything is executed. Stage isolation re-runs a single partition with an intermediate hash after each stage; it costs one re-run of one partition, and it eliminates every stage before the first divergence. Environment variation re-runs that one stage under systematically varied conditions — one worker, one thread, the released image, the released native libraries — and costs one short re-run per experiment. A footer records the ordering rule and the reason for it: each axis is cheaper than the next and eliminates more, so attempting them out of order spends the expensive experiments answering a question the free comparison would have answered. Free, then cheap, then expensive — and each one eliminates more 1 · spatially compare per-partition digests 0 re-runs one partition differs → local to it all differ → environment or global derivation a block differs → something partitions the work the same way none differ, artifact does → the merge 2 · by stage hash the intermediate after each stage 1 partition re-run eliminates every stage before the first divergence needs a canonical serialisation, or the hash is meaningless 3 · by environment vary one axis per run 1 short re-run each workers, threads, hash seed, image digest, native libraries the first experiment that matches names the cause Attempt them in this order. Each axis is cheaper than the next and eliminates more — out of order, the expensive experiments answer a question the free comparison had already settled.
Attempt the axes in this order: each is cheaper than the next and eliminates more, and the first one costs no re-runs at all.

Minimal Reproducer: Hash at the Partition Level

The single most useful thing to add to a generator, and the thing that makes every subsequent step possible, is a per-partition hash written alongside the artifact.

python
import hashlib
import json
from pathlib import Path


def partition_digests(artifact_dir: Path) -> dict[str, str]:
    """One digest per partition file, keyed by the partition's identity."""
    out = {}
    for p in sorted(artifact_dir.glob("tile_*.parquet")):
        h = hashlib.blake2b(digest_size=16)
        h.update(p.read_bytes())
        out[p.stem] = h.hexdigest()
    return out


released = json.loads(Path("release-2026Q2/partitions.json").read_text())
repro = partition_digests(Path("repro"))

differing = sorted(k for k in released if released[k] != repro.get(k))
missing = sorted(set(released) - set(repro))
extra = sorted(set(repro) - set(released))
print(f"{len(differing)} differing, {len(missing)} missing, {len(extra)} extra")
print("first few differing:", differing[:5])
Four canonicalisation rules for intermediate hashes, and the false diagnosis each prevents Four rows. Column ordering: a frame whose columns arrive in dictionary order hashes differently on two runs that produced identical data; canonicalise by sorting the column names before serialising, and without it the investigation is sent into whichever stage last added a column. Float formatting: the shortest round-trippable representation differs between library versions, so identical values serialise differently; canonicalise by writing a fixed number of significant digits, or by hashing the raw bytes of the array rather than a text form, and without it every stage appears to diverge at once. Index presence: a reset or preserved index changes the serialisation without changing the data; canonicalise by dropping the index before hashing, and without it a reordering upstream looks like a data change. Metadata: file-level attributes such as a creation timestamp or a library version string are written into the container rather than the data; canonicalise by hashing the columns rather than the file, and without it every hash differs on every run and the whole technique is abandoned as useless. A closing note records the failure mode this table exists to prevent: an intermediate hash that is itself non-deterministic is worse than no intermediate hash, because it actively sends the investigation into the wrong stage. A non-deterministic intermediate hash is worse than none at all Source canonicalisation false diagnosis without it cost column ordering sort column names before serialising points at whichever stage added a column free float formatting fixed significant digits, or hash raw bytes every stage appears to diverge at once free index presence drop the index before hashing an upstream reorder looks like a data change free container metadata hash the columns, not the file every hash differs every run — the technique is abandoned one helper All four are cheap and all four are usually discovered the hard way — during an investigation, when the intermediate hashes disagree and the data does not.
An intermediate hash only helps if it is deterministic — these four are the sources of spurious difference, and all four are cheap to remove.

The shape of that output is the diagnosis, before anything is re-run:

  • One or a few partitions differ. The cause is local to those partitions — a data-dependent code path, an input asset that only they touch, a numerical edge case at a specific coordinate.
  • Every partition differs. The cause is global: environment, library version, or a seed derivation that changed for everything at once.
  • A contiguous block differs. Look for something that partitions the work the same way — a worker assignment, a shard boundary, an input tile that covers exactly that block.
  • The partitions match and the merged artifact does not. The merge is the cause. Almost always an unordered reduction or a non-deterministic write order.

That last case is worth stressing because it is the one people find last: every tile is byte-identical, so the generator is exonerated, and the defect is in twenty lines of merge code nobody thought to check.

Fix: Narrow, Then Isolate

Narrow spatially, then re-run one partition

python
target = differing[0]
row, col = (int(x) for x in target.removeprefix("tile_").split("_"))

one = generate_tile(manifest, row=row, col=col, out=Path("audit/one"))
print("isolated tile matches release:",
      partition_digests(Path("audit/one"))[target] == released[target])

If the isolated tile matches the release but differed in the full reproduction, the divergence is not in the tile’s generation at all — it is in something the full run does that the isolated run does not, which is a short list: work distribution, shared state, and the merge.

If the isolated tile differs from the release too, the divergence is inside that tile’s generation, and stage isolation is next.

Isolate by stage

python
STAGES = ("sample", "attributes", "topology", "privacy", "write")


def staged_digests(manifest: dict, row: int, col: int) -> dict[str, str]:
    """Hash the intermediate after each stage, so the first divergence is visible."""
    state = None
    out = {}
    for stage in STAGES:
        state = run_stage(stage, manifest, state, row=row, col=col)
        out[stage] = hashlib.blake2b(canonical_bytes(state), digest_size=16).hexdigest()
    return out


ref = json.loads(Path("release-2026Q2/stage-digests.json").read_text())[target]
now = staged_digests(manifest, row, col)
first_bad = next((s for s in STAGES if ref[s] != now[s]), None)
print("first diverging stage:", first_bad)

canonical_bytes is doing quiet work: hashing an in-memory structure requires a canonical serialisation, or the hash reflects dict ordering and float formatting rather than the data. Sort the columns, fix the float repr, and write it once as a utility — an intermediate hash that is itself non-deterministic is worse than no intermediate hash, because it sends the investigation into the wrong stage.

Vary the environment, one axis at a time

By this point the divergence is localised to one stage of one partition, and the remaining question is which environment difference produces it. Vary one thing per run and record the result:

python
EXPERIMENTS = [
    ("workers=1",           {"jobs": 1}),
    ("threads=1",           {"env": {"OMP_NUM_THREADS": "1"}}),
    ("hashseed=0",          {"env": {"PYTHONHASHSEED": "0"}}),
    ("released image",      {"image": manifest["environment"]["image_digest"]}),
    ("released PROJ only",  {"proj": manifest["environment"]["proj"]}),
]
for label, override in EXPERIMENTS:
    d = run_stage_isolated(first_bad, manifest, row, col, **override)
    print(f"{label:<20} {'MATCHES' if d == ref[first_bad] else 'differs'}")

The first experiment that produces a match names the cause. In practice one of the last two matches far more often than the first three, and the reason is that native library versions are the part of the environment least likely to have been pinned and most likely to change what geometry code produces.

Distribution of reproduction-divergence causes, with the confirming experiment for each Causes run along the horizontal axis, ordered by how often they are the answer. Native library version differences — PROJ or GEOS built into a different wheel — are the most common single cause, and they are confirmed by pinning the released versions and re-running one stage. An unordered reduction is next, and it is distinctive because every partition matches while the merged artifact does not, so the free spatial comparison identifies it before anything is re-run. Input drift — an asset corrected since the release — follows, and it is caught by the digest check during restoration rather than by any experiment. A schedule-dependent seed derivation is next, confirmed by regenerating one partition alone. Thread-count sensitivity and hash-seed sensitivity are the remaining two, each confirmed by setting the single variable and re-running. Above each bar is the narrowing axis that locates it. The pattern the chart is making is that the two most common causes are both found by the cheapest checks — one by the digest verification during restoration and one by a free comparison of partition hashes — so an investigation that starts by re-running the pipeline has skipped past the answer more often than not. The two most common causes are found by the two cheapest checks 0% 10% 20% 30% 40% share of investigations (%) 31% native library version environment pin PROJ/GEOS, re-run one stage 24% unordered reduction spatial (free) partitions match, artifact does not 18% input drift restoration (free) the digest check refuses 14% schedule-dependent seed spatial regenerate one partition alone 8% thread-count sensitivity environment set threads=1, re-run 5% hash-seed sensitivity environment set PYTHONHASHSEED, re-run An investigation that starts by re-running the pipeline has skipped past the answer more often than not: the top two causes are found by the digest check during restoration and by a free comparison of partition hashes.
Where divergences are actually found, by the axis that localised them — and the experiment that names each cause.

Verification Step: Turn the Finding Into a Gate

An audit that ends with an explanation and no gate will be repeated. Whatever the cause turns out to be, close it with an assertion that runs on every release:

python
def test_release_is_reproducible_at_partition_level(tmp_path):
    """Regenerate three partitions and compare against the recorded digests."""
    manifest = json.loads(Path("release/manifest.json").read_text())
    recorded = json.loads(Path("release/partitions.json").read_text())
    for key in sample_partitions(recorded, k=3, seed=manifest["root_seed"]):
        row, col = parse_key(key)
        got = generate_tile(manifest, row=row, col=col, out=tmp_path / key)
        assert partition_digests(tmp_path / key)[key] == recorded[key], key

Sampling three partitions rather than all of them keeps the check affordable per release; deriving which three from the root seed keeps it deterministic, so a failure is reproducible in its own right.

Recording the Investigation

An audit produces two things, and the second one is the one that matters in six months.

The first is the finding: this stage, in this partition, diverged because of this environment difference. That gets fixed and the fix gets a gate.

The second is the elimination record — everything the investigation ruled out on the way. It is worth writing down for the same reason a medical differential is worth writing down: the next person to see a similar symptom starts from what has already been excluded rather than from scratch. A short structured note is enough:

python
audit = {
    "release": manifest["artifact"]["path"],
    "symptom": "artifact hash mismatch, all partitions differ",
    "eliminated": [
        "input drift — all digests verified at restoration",
        "seed derivation — single partition reproduces in isolation",
        "reduction order — per-partition hashes also differ",
    ],
    "cause": "GEOS 3.12.1 vs recorded 3.11.2 — wheel rebuilt upstream",
    "gate_added": "assert shapely.geos_version_string == manifest.environment.geos",
}

The eliminated list is what makes this reusable. A future audit that sees the same symptom can start by checking whether the same eliminations hold, and in practice the same few causes recur across a platform because they come from how it is built rather than from what it generates. Three or four of these notes are usually enough to make the next investigation an afternoon.

Edge Cases & Gotchas

The divergence is one bit in one float. Usually a fused multiply-add or a vectorised path taken on one CPU and not another. Pin the instruction-set baseline in the container build rather than chasing the arithmetic; the fix is a build flag, not a code change.

Nothing reproduces the divergence. The variation is not in the environment axes you varied. The remaining candidates are wall-clock or entropy sources reached indirectly — a library that seeds itself, a UUID in a temporary path that ends up in an output column, a “last modified” field written into metadata. Grep the output for anything that changes between two runs of the same command before varying anything further.

The audit itself is not reproducible. If the experiment harness derives its own seeds or writes to a shared directory, its results will drift between attempts. Give the audit its own manifest, and treat it as a run like any other.

The divergence is in a stage that no longer exists. The generator has been refactored since the release. That makes the release un-auditable with today’s code, and the finding is an argument for retaining generator versions alongside artifacts rather than a defect to chase.