Tracing a Downstream Defect Back to Its Generator

A consumer reports that their model’s behaviour changed. They are using “the latest” synthetic release, they did not change anything on their side, and the release passed every gate on the way out.

Part of Artifact Versioning & Lineage: this is the investigation that registry exists to make possible, and the procedure for running it in the order that eliminates the most per step.

Root Cause: Three Things Change and Only One Is Yours

Before touching the generator, establish which of three things actually moved. In practice the distribution is roughly even, and skipping the first two costs days.

The consumer’s pipeline changed. A library upgrade, a config edit, a different join. This is settled in one step by asking them to re-run their current pipeline against the previous release, which the registry can hand them by hash. If the behaviour reverts, the release is exonerated and the investigation moves to their side.

The release changed for a legitimate reason. A new region was added, a fallback rule fired more often, an input was updated. The output genuinely differs and no defect exists — but the consumer was not told, and “no defect” is not the same as “nothing to do”.

The generator changed. Something in the pipeline produces different output from the same inputs, and that is a defect.

Resolution of downstream defect reports, with the step that settles each A single horizontal bar covers three thousand simulated reports, split into four segments. The largest is that the consumer's own pipeline changed, which is settled by one query: re-run their current code against the previous release, which the registry can hand them by hash. The second largest is that the release legitimately changed — a new region, a fallback firing more often, an updated input — which is settled by diffing the lineage of the two releases, again one query. Together those two are the majority, and neither requires touching the generator. The third is a genuine generator change, which needs a bisection over the release chain followed by a diff of the generator commits. The smallest is that the release does not reproduce at all, which is a different and worse finding and moves to the reproduction audit. Each segment carries its settling step. The conclusion drawn underneath is that most of a defect investigation's value comes from the two cheapest steps, and both of them are queries against a record that has to have been written at the time. The two largest branches are each one query 36% 32% 25% the consumer's pipeline changed — 36% settled by: one query: re-run their code on the previous release the release legitimately changed — 32% settled by: one query: diff the lineage of the two releases the generator changed — 25% settled by: bisect the release chain, then diff the generator commits the release does not reproduce — 6% settled by: the reproduction audit — a different, worse finding 3,000 simulated reports, fixed seed. Most of the value comes from the two cheapest steps, and both are queries against a record that had to have been written at the time.
Across simulated investigations: the first two branches account for the majority, and both are settled by a single query the registry can answer.

Prerequisite Check: Establish What They Are Actually Using

python
def resolve_consumer_release(registry, consumer: str) -> dict:
    """The consumption log answers this in one query — and nothing else does."""
    events = registry.consumption(consumer=consumer, limit=20)
    if not events:
        raise SystemExit(
            f"{consumer} has no recorded consumption — they are bypassing resolve(). "
            f"Find out how before continuing; the rest of this procedure assumes the record."
        )
    return {
        "current": events[0]["artifact_sha"],
        "previous": next((e["artifact_sha"] for e in events
                          if e["artifact_sha"] != events[0]["artifact_sha"]), None),
    }

If this raises, stop and fix that first. Every step below reads from the record, and an investigation that starts by guessing which release a consumer has is an investigation that will end by disagreeing about it.

Fix: Bisect Over Releases, Then Over Inputs

1 — Confirm the release is implicated at all

python
def confirm_release_implicated(consumer_harness, shas: dict) -> bool:
    """Re-run the consumer's current code against the previous release."""
    now = consumer_harness.run(shas["current"])
    before = consumer_harness.run(shas["previous"])
    return now.metric != before.metric

One run, and it splits the space in half. If the metric is the same on both, the consumer’s code changed and the release is not involved.

2 — Bisect the release range

If several releases sit between the last known-good and the first known-bad, bisect rather than diffing the endpoints. The registry’s supersession chain gives the ordering, and each step is one consumer run:

python
def bisect_releases(registry, name: str, good: str, bad: str, harness) -> str:
    chain = registry.chain(name, from_sha=good, to_sha=bad)     # ordered oldest → newest
    lo, hi = 0, len(chain) - 1
    while hi - lo > 1:
        mid = (lo + hi) // 2
        if harness.run(chain[mid]["artifact_sha"]).metric == harness.run(good).metric:
            lo = mid
        else:
            hi = mid
    return chain[hi]["artifact_sha"]        # the first release exhibiting the change

3 — Diff the implicated release against its predecessor, by lineage

python
def diff_lineage(registry, a_sha: str, b_sha: str) -> dict:
    a, b = registry.get(a_sha), registry.get(b_sha)
    inputs_a, inputs_b = a["input_shas"], b["input_shas"]
    return {
        "generator": (a["generator_version"], b["generator_version"]),
        "inputs_changed": sorted(k for k in set(inputs_a) | set(inputs_b)
                                 if inputs_a.get(k) != inputs_b.get(k)),
        "contract": (a["contract_sha"], b["contract_sha"]),
        "privacy": (a["privacy_ledger_row"], b["privacy_ledger_row"]),
    }

Three of those four fields are usually identical, and whichever one is not is the cause. That is the entire payoff of recording lineage at promotion time: the question “what changed” becomes a dictionary comparison rather than an archaeology exercise.

Lineage diff between two adjacent releases, with the single differing field marked Two columns show the registry records for the last known-good release and the first release exhibiting the reported change. Row by row: the artifact hashes differ, which is expected and carries no information. The versions differ by a minor bump. The generator versions are identical, so no code changed. The contract hashes are identical, so no declaration changed. The privacy ledger rows differ only in their identifier, with the same epsilon, so the privacy accounting did not move. Four input digests are listed; three match and one — the boundary layer — differs, and it is marked. That single differing field is the cause, and locating it took a dictionary comparison rather than an investigation. Beneath the two columns, the next step is stated: re-run the newer release's manifest with the older version of that one input, and if the reported behaviour reverts, the finding is complete and it is not a generator defect — an input changed and the release faithfully reflected it. Three fields match, one does not — and the one that does not is the cause last known good first showing the change artifact_sha 9c1f…a03e 4b77…e15c expected — carries no information version 2026.3.1 2026.4.0 a minor bump generator_version gen@1f4c9d2 gen@1f4c9d2 no code changed contract_sha e21a…77b0 e21a…77b0 no declaration changed privacy_ledger_row ldg-8841 (ε 0.5) ldg-8902 (ε 0.5) same spend, new row input: population_raster aa19…c4 aa19…c4 input: boundary_layer 77bd…19 0e34…f2 ← the cause input: network_extract 31c8…6a 31c8…6a input: covariate_stack d509…8b d509…8b Next: re-run the newer manifest with the older boundary layer. If the behaviour reverts, no generator defect exists.
The lineage diff between two adjacent releases: three fields match, one does not, and the one that does not is the cause.

4 — Reproduce the change in isolation

python
def isolate(registry, bad_sha: str, changed_input: str) -> str:
    """Re-run the bad release's manifest with the previous version of one input."""
    manifest = registry.manifest(bad_sha)
    good_input = registry.get(registry.get(bad_sha)["supersedes"])["input_shas"][changed_input]
    manifest["inputs"][changed_input] = good_input
    return generate(manifest, out=Path("audit/isolated"))

If swapping that one input back reproduces the previous behaviour, the finding is complete and it is not a generator defect — an input changed and the release faithfully reflected it. If it does not, the generator version is the remaining candidate, and the diff between the two generator commits is now a small, bounded thing to read.

Verification Step: Close With a Fixture, Not a Fix

python
def test_regression_fixture(fixture_dir):
    """Whatever the cause, lock the behaviour that changed."""
    manifest = json.loads((fixture_dir / "manifest.json").read_text())
    out = generate(manifest, out=fixture_dir / "actual")
    expected = json.loads((fixture_dir / "expected-metrics.json").read_text())
    got = consumer_metric(out)
    assert abs(got - expected["metric"]) < expected["tolerance"], (got, expected)

The fixture is the deliverable of the investigation, and it is worth more than the fix. It carries a manifest small enough to run in CI, the metric the consumer actually watches, and the tolerance that was agreed — so the next change that moves it is caught by the producer rather than reported by the consumer.

Two properties make a fixture useful rather than decorative. It must be small: a fixture that takes twenty minutes will be skipped, so cut the extent down until it runs in seconds while still exhibiting the behaviour. And it must assert the consumer’s metric, not an internal one — the whole point is that the internal gates passed.

Edge Cases & Gotchas

The consumer cannot re-run against an old release. Their harness may not be parameterised by version. Getting them to that point is itself worth doing, because without it every future report is unattributable, and it is usually a small change to a config path.

Five properties of a useful regression fixture and the failure of omitting each Five rows. A small extent, cut down until the fixture runs in seconds while still exhibiting the behaviour; without it the fixture takes minutes, gets marked slow, and is skipped within a month. The consumer's own metric rather than an internal one; without it the fixture asserts something the internal gates already covered, and the whole point was that those passed. An agreed tolerance, negotiated with the consumer rather than chosen by the producer; without it the fixture either fails on noise or passes through the regression it was written for. A pinned manifest with its own seed and inputs; without it the fixture drifts with whatever the pipeline currently defaults to and stops testing the thing it captured. And a recorded cause — one sentence naming what was found; without it the fixture is a mysterious assertion nobody dares change, which is how a suite accumulates tests that are never updated and never trusted. A footer records the observation that follows: the fixture is the deliverable of the investigation, and it is worth more than the fix, because the fix addresses one defect and the fixture addresses the class. The fixture is the deliverable — the fix addresses one defect, it addresses the class small extent runs in seconds, still exhibits the behaviour WITHOUT IT marked slow, then skipped within a month the consumer's metric not an internal one WITHOUT IT asserts what the internal gates already covered an agreed tolerance negotiated, not chosen by the producer WITHOUT IT fails on noise, or passes the regression it was written for a pinned manifest its own seed, its own inputs WITHOUT IT drifts with the pipeline's current defaults a recorded cause one sentence naming what was found WITHOUT IT a mysterious assertion nobody dares change A suite of five small fixtures, each carrying a sentence about what it caught, is worth more than a hundred assertions nobody can explain — and it is the only artifact from an investigation that keeps paying.
The fixture outlives the investigation, which is why its properties are worth getting right.

The bisection is not monotone. Two independent changes in the range, one of which was later reverted. The bisection converges on the wrong release and the diff looks innocuous. The tell is that the metric moves back and forth along the chain rather than once, which is visible if the harness records every run rather than only the branch it takes.

The changed input is upstream of another artifact. The lineage walk recurses, and the changed thing may be several levels up — a boundary file feeding a tessellation feeding the release. That is the case the transitive closure exists for, and it is why a lineage graph beats a lineage string.

Nothing changed and the behaviour still differs. Then the release is not reproducible, and the investigation moves to auditing a run that cannot be reproduced. This is a different and worse finding than any of the above, because it means two releases with identical recorded inputs are not identical.

What to Tell the Consumer, and When

An investigation has an audience, and the reporting is part of the work rather than an afterthought.

At the start, confirm what they are actually using and say so. “You are on 2026.4.0, promoted on the fourteenth, superseding 2026.3.1” settles more than it looks: it establishes that the producer can answer the question at all, and it frequently prompts the consumer to notice that they upgraded at the same moment their behaviour changed.

When the release is exonerated, say that plainly and give them the evidence — their code against the previous release produced the same result. This is not a deflection if it comes with the hash and the harness output, and it usually shortens their own investigation considerably.

When the release legitimately changed, name the change and its cause: an input was updated, a region was added, a fallback started firing. This is the case where “no defect” is genuinely the answer and it is also the case where a release note should have existed and did not. Adding the change to the release record afterwards is worth doing even though it is late.

When it is a generator defect, report the finding with the fixture rather than the fix. A consumer who has been told “we fixed it” has no way to know whether it will recur; one who has been told “here is a test that now runs on every release, asserting your metric within your tolerance” has.

When nothing is found, say that too. An investigation that ends without a cause is a real outcome and it happens: an intermittent difference in the consumer’s environment, a metric with more run-to-run variance than anybody realised, a change neither side recorded. Closing it explicitly, with what was eliminated, is far better than letting it fade — because the next report of the same symptom then starts from the elimination list rather than from scratch.