Reproducing a Release From Its Manifest

Somebody needs last quarter’s synthetic release regenerated — to compare against this quarter’s, to investigate a defect a consumer reported, or because an auditor asked whether it can be done at all — and you have the manifest and nothing else.

Part of Seed Management & Run Reproducibility: that page covers how to build a pipeline whose runs can be reproduced. This one is the procedure for actually doing it, in the order that fails fastest, and the verification that turns a regenerated file into evidence.

Root Cause: Why It Usually Fails on the First Attempt

A manifest records the state the run depended on. Reproducing it means restoring that state, and the reason a first attempt usually fails is that restoration has an order. Each of these depends on the one before it, and attempting them out of order produces a failure whose message points at the wrong thing:

  1. The container. Everything below runs inside it. Restore it first, by digest, and never by tag.
  2. The environment variables. PYTHONHASHSEED and the thread counts change results and appear in no dependency list, so a run that restores packages but not these fails with no visible cause.
  3. The packages. Install the resolved list, not the requirement list, and verify the native library versions afterwards rather than trusting the wheel names.
  4. The inputs. Fetch each asset and check its digest before use. An input that has been “corrected” since the release is the single most common reason a reproduction diverges, and it diverges silently.
  5. The seed derivation. Replay it from the recorded root and rule; do not draw fresh streams.

Attempting step five with step four unverified produces an artifact that differs and gives no indication why. Attempting step three inside the wrong container produces a native library mismatch that presents as a geometry difference of a few micrometres, which then propagates into a completely different hash.

The five restoration steps, their checks, and the misleading symptom of skipping each Five steps run top to bottom. One, restore the container by digest: everything else runs inside it; the check is that the pulled digest equals the recorded one; skip it and a native library mismatch presents as a geometry difference of a few micrometres that propagates into a completely different hash. Two, restore the environment variables: the hash seed and the thread counts; the check is that each is set before the interpreter starts; skip them and the run fails with no visible cause at all, because neither appears in any dependency list. Three, install the resolved package list with dependency resolution disabled; the check is that the PROJ and GEOS version strings match the recorded ones rather than that the wheel names do; skip it and a transitive dependency is silently re-resolved. Four, fetch every input and verify its digest before use; the check is a refusal on mismatch; skip it and an input corrected since the release makes the reproduction a different run that happens to share code. Five, replay the seed derivation from the recorded root; the check is that no stream is drawn from the clock; skip it and nothing reproduces. A footer records why the order is worth following: each step is cheaper than the one after it and its failure is unambiguous, whereas the same failure surfaced later is not. Each step's failure presents as a defect in a later one 1 · container, by digest everything below runs inside it CHECK pulled digest == recorded digest IF SKIPPED a native mismatch surfaces as a micrometre geometry difference 2 · environment variables PYTHONHASHSEED, thread counts CHECK set before the interpreter starts IF SKIPPED no visible cause — neither appears in any dependency list 3 · resolved packages install with --no-deps CHECK PROJ and GEOS version strings match IF SKIPPED a transitive dependency is silently re-resolved 4 · inputs, by digest fetch, then verify before use CHECK refuse on mismatch, do not warn IF SKIPPED a corrected input makes it a different run sharing code 5 · seed derivation replay from the recorded root CHECK no stream drawn from the clock IF SKIPPED nothing reproduces, and the message points elsewhere Each step is cheaper than the one after it and its failure is unambiguous — the same failure surfaced two steps later is not.
Restoration has an order, and each step's failure presents as a defect in a later one — which is why the checks are cheapest to run in this sequence.

Prerequisite Check: Confirm the Manifest Is Complete

Before restoring anything, verify the manifest can support a reproduction at all. A manifest missing any of these fields cannot, and finding that out now costs a minute rather than an afternoon.

Manifest field presence against the cost of each field's absence Eight manifest fields run down the vertical axis, ordered by how often they are present in a release manifest. The root seed and a requirements file are recorded almost always, because they are what a tutorial mentions. The artifact hash — without which a reproduction has no definition of success — is present about two thirds of the time. The resolved package list, as distinct from the requirement list, is present about half the time. Input digests, the container digest, the native PROJ and GEOS versions, and the hash seed together with the thread counts become progressively rarer, and the last of those is recorded roughly one time in nine. Beside each bar is what its absence makes impossible. The pattern is an inverse one: the fields most often recorded are the ones whose absence is easiest to work around, and the fields most often omitted are the ones whose absence ends the reproduction outright, because they are invisible in every diff a reviewer would think to run. The note underneath draws the practical conclusion — the manifest schema should be validated at promotion, not discovered at reproduction, since every one of these is trivial to record and impossible to recover later. The fields most often omitted are the ones whose absence ends the attempt root seed 97% without it: nothing can be replayed requirements file 91% without it: the resolved set is unknown artifact hash 68% without it: there is no definition of success resolved package list 54% without it: a transitive bump goes unnoticed input digests 41% without it: an edited input is invisible container digest 33% without it: the base image drifts underneath PROJ / GEOS versions 19% without it: geometry differs with no diff hash seed + threads 11% without it: results change with nothing recorded Validate the manifest schema at promotion rather than discovering the gaps at reproduction: every field here is trivial to record at the time and impossible to recover afterwards.
Field presence against the cost of absence: the fields teams record are the ones a tutorial mentions, and the fields that block a reproduction are the ones left out.
python
import json
from pathlib import Path

REQUIRED = {
    "root_seed", "derivation", "environment", "inputs", "artifact",
}
REQUIRED_ENV = {"packages", "proj", "geos", "hash_seed", "threads", "image_digest"}


def check_manifest(path: Path) -> list[str]:
    m = json.loads(path.read_text())
    gaps = sorted(REQUIRED - m.keys())
    gaps += [f"environment.{k}" for k in sorted(REQUIRED_ENV - m.get("environment", {}).keys())]
    if not m.get("inputs"):
        gaps.append("inputs (empty)")
    if "sha" not in m.get("artifact", {}):
        gaps.append("artifact.sha — nothing to compare a reproduction against")
    return gaps


gaps = check_manifest(Path("release-2026Q2/manifest.json"))
print("cannot reproduce; missing:", gaps) if gaps else print("manifest is complete")

If artifact.sha is missing, stop. Without it there is no definition of success, and a regenerated file that looks plausible proves nothing.

Fix: The Restoration Procedure

Restore the container by digest

bash
# The tag in the manifest is informational. The digest is the identity.
DIGEST=$(jq -r '.environment.image_digest' release-2026Q2/manifest.json)
docker pull "ghcr.io/example/spatial-gen@${DIGEST}"
docker run --rm -it \
  -v "$PWD/release-2026Q2:/work" \
  "ghcr.io/example/spatial-gen@${DIGEST}" bash

If the digest is no longer available in the registry, that is a finding in its own right and should be recorded: the release is no longer reproducible, and the fix is a retention policy on generation images rather than anything in the pipeline.

Restore the environment and verify the native libraries

python
import os
import subprocess
import sys

env = json.loads(Path("/work/manifest.json").read_text())["environment"]

os.environ["PYTHONHASHSEED"] = env["hash_seed"]
os.environ.update({k: v for k, v in env["threads"].items() if v})

subprocess.run([sys.executable, "-m", "pip", "install", "--no-deps",
                *env["packages"]], check=True)

import pyproj, shapely
assert pyproj.proj_version_str == env["proj"], (
    f"PROJ {pyproj.proj_version_str} != recorded {env['proj']}"
)
assert shapely.geos_version_string == env["geos"], (
    f"GEOS {shapely.geos_version_string} != recorded {env['geos']}"
)

--no-deps matters. Installing the resolved list with dependency resolution enabled lets pip re-resolve transitive requirements and quietly install something the original run did not have.

PYTHONHASHSEED has to be set before the interpreter starts to take effect, so in practice this block belongs in an entrypoint that re-executes the process — setting it from inside a running interpreter changes nothing.

Resolve every input by digest, and refuse a mismatch

python
def fetch_inputs(manifest: dict, dest: Path) -> None:
    for rel, expected in sorted(manifest["inputs"].items()):
        path = dest / rel
        path.parent.mkdir(parents=True, exist_ok=True)
        fetch_from_archive(rel, path)          # your object store, CAS, or backup
        actual = asset_digest(path)
        if actual != expected:
            raise SystemExit(
                f"input drift: {rel}\n  recorded {expected}\n  fetched  {actual}\n"
                f"  the release cannot be reproduced from the current asset"
            )

Raising here rather than warning is the whole point. An input that has changed since the release means the reproduction would be a different run that happens to use the same code, and reporting its hash mismatch later as a pipeline defect wastes the next several hours.

Replay the derivation and generate

python
manifest = json.loads(Path("/work/manifest.json").read_text())
result = generate(
    root_seed=manifest["root_seed"],
    inputs=Path("/work/inputs"),
    out=Path("/work/repro"),
)

Nothing here re-derives anything from the clock, the hostname or the worker count. If generate accepts a seed at all it accepts this one, and if it accepts anything else — a --jobs flag that changes the partitioning, say — that flag is part of the manifest or the pipeline is not reproducible.

Verification Step: Compare, and Record the Comparison

python
def verify(repro: Path, manifest: dict) -> None:
    actual = asset_digest(repro)
    expected = manifest["artifact"]["sha"]
    verdict = {
        "release": manifest["artifact"]["path"],
        "expected_sha": expected,
        "actual_sha": actual,
        "reproduced": actual == expected,
    }
    Path("/work/reproduction-report.json").write_text(
        json.dumps(verdict, indent=2, sort_keys=True)
    )
    if not verdict["reproduced"]:
        raise SystemExit("artifact hash mismatch — see reproduction-report.json")

Write the report whether or not it succeeded. A failed reproduction is evidence too, and it is the input to auditing a run that cannot be reproduced; a failure that leaves no artifact behind has to be re-run before it can be investigated.

Expected time to locate a divergence, by investigation method and partition count The horizontal axis is the number of partitions in the release, on a log-2 scale from sixteen to sixteen thousand. The vertical axis is the expected time in minutes to locate the source of a divergence. The first curve is the method used when no per-partition digests exist: re-run the whole pipeline, inspect the output, form a hypothesis and re-run again. It does not depend on the partition count — it depends on the pipeline runtime — so it is a flat, expensive line. The second is bisection over recorded partition digests: comparing digests is free, and the only re-runs needed are the handful of single-partition regenerations that confirm the hypothesis, so the cost grows with the logarithm of the partition count and stays small across the whole range. The gap between the curves at the right edge is printed. The note underneath states the precondition that makes the second curve available: partition digests are only useful if the seed derivation was keyed on partition identity, because otherwise a single partition cannot be regenerated in isolation and there is nothing to compare. Bisection is only available if the seeds were derived from partition identity 16 64 256 1,024 4,096 16,384 0 50 100 150 partitions in the release (log scale) expected time to locate the divergence (min) re-run and inspect: 126 min bisect over digests: 6 min 22× re-run the whole pipeline and inspect bisect over recorded partition digests A 42-minute pipeline against 24-second single-partition regenerations. The second curve is only available if the seed derivation was keyed on partition identity — otherwise a partition cannot be regenerated alone and there is nothing to compare.
When the hashes disagree, comparing at the partition level narrows the cause logarithmically — provided the seed derivation was keyed on partition identity in the first place.

What a Successful Reproduction Actually Proves

It is worth being precise about this, because a reproduction report is often quoted as though it settled more than it does.

A matching hash proves that the recorded inputs, run through the recorded environment with the recorded seed, produce the recorded artifact. That is exactly the claim, and it is a strong one: it means the release is not a one-off, that nothing undeclared influenced it, and that a reviewer can inspect the process rather than taking the output on trust.

It does not prove the release is correct. A generator with a bug reproduces its bug perfectly, and a reproduction is silent on whether the statistics are faithful, whether the privacy budget was computed properly, or whether the contract was the right contract. Those are the validation gates’ job, and a release that reproduces and fails them is reproducibly wrong.

Nor does it prove the release will reproduce next year. It proves it reproduced today, in an environment that was restorable today. The registry entry for the image digest, the retention of the input assets and the availability of the generator version are all live dependencies, and any one of them can lapse quietly. That is why the reproduction report is worth writing to the registry rather than to a terminal: a record of when a release was last successfully reproduced is the only early warning that the ability to do so has decayed.

Edge Cases & Gotchas

The release predates the manifest schema. Older releases often carry a partial record — a seed and a requirements file, and nothing else. They are not reproducible, and saying so plainly is better than producing a file that “looks right”. What is worth doing is recording, once, which releases fall on which side of the line, so the question is answered from a table rather than re-investigated each time.

The reproduction succeeds and a consumer still sees a difference. The consumer is comparing something other than the artifact — a derived index, a cached tile pyramid, a database load with its own ordering. Compare hashes at the artifact boundary first, and only then walk forward through whatever transformed it.

The generator has been fixed since the release. Then reproducing the release requires the generator version the manifest names, which means the generator has to be versioned and retained like any other input. Reproducing a release with today’s generator is a useful experiment and is not a reproduction; label it accordingly or it will be quoted as one.

Reproduction spends privacy budget. It does not, provided it genuinely reproduces — the same artifact is not a new observation of the population. A “reproduction” that draws fresh noise is a second release and must be debited, which is one more reason the hash comparison is not optional.