Seed Management & Run Reproducibility for Spatial Generation

Reproducibility is not a property a generation pipeline has; it is a property a specific run has, and it survives only if the things that determined it were written down while the run was happening. This page is part of Synthetic Spatial Data Architecture & Fundamentals, and it covers the sub-problem every other page in that area quietly depends on: given a release and its manifest, can you produce that release again, on a different machine, a year later, byte for byte — and can you prove it before anybody asks?

The answer is almost never “yes, because we set a seed.” A seed fixes one source of variation among five or six, and the other sources are precisely the ones that differ between the machine a pipeline was written on and the machine it eventually runs on. What makes a run reproducible is a derivation discipline: a single recorded root, a rule that turns it into every downstream stream, and a manifest that pins everything the derivation cannot control.

Problem Framing: Five Sources, and a Conjunction

The failure this page prevents is the one that arrives as a question rather than an error. Somebody asks why a model trained on last quarter’s synthetic release behaves differently from one trained on this quarter’s, and the honest answer turns out to be that nobody knows, because the older release cannot be regenerated to compare.

Reproducibility is a conjunction. Every one of the following must hold, and the run is non-reproducible if any single one fails:

  • The random stream is derived, not shared. A single global generator consumed by several stages, or by several parallel workers, produces values that depend on the order in which those consumers happened to run. That order is a property of the scheduler, not of the pipeline.
  • The toolchain is pinned to resolved versions, not requested ones. geopandas>=0.14 is a request. What ran was one specific version, and a patch release of GEOS can change which of two coincident vertices survives a repair.
  • The container is pinned by digest, not by tag. A tag is a mutable pointer. The image that python:3.11-slim resolves to today is not the image it resolved to six months ago.
  • Reductions are ordered. Floating-point addition is not associative, so summing partial results in completion order produces a different total on every run. The per-partition outputs can be byte-identical while the aggregate statistics are not.
  • Every input asset is content-addressed. A network extract, a covariate raster, a boundary file: each one is an input, and “the current version” is not an identifier.
Reproduction rate as five sources of run-to-run variation are closed in turn Six bars read left to right as discipline accumulates. The first, with nothing pinned, reproduces almost never. Pinning dependencies, then the container tag, then ordering the reduction each improve matters only slightly, because the two remaining sources — a shared mutable generator across workers and a clock-derived seed — dominate everything else so completely that the bar barely leaves the axis. Even with four of the five closed, fewer than two runs in a hundred reproduce. Only when the last source is closed does the rate jump to one hundred per cent. Each bar prints its measured rate. The shape is the point: because the conditions compose multiplicatively, partial discipline buys almost nothing, and the two sources that dominate are the two that are cheapest to fix and least often thought of as reproducibility problems at all. Partial discipline buys almost nothing — the conditions multiply 0% 25% 50% 75% 100% re-runs reproducing byte-identical output 0.3% 0.2% 0.3% 0.6% 1.7% 100.0% nothing pinned + unpinned patch dependencies + floating container tag + unordered reduction + shared mutable RNG across workers + clock-derived seed 20,000 simulated re-runs per configuration, fixed seed, sources closed in the order teams usually think of them. The two that dominate — a shared generator and a clock-derived seed — are also the two cheapest to fix, and neither is usually filed as a reproducibility problem.
Measured over 20,000 simulated re-runs: reproducibility survives only when every source is closed, and the two most commonly overlooked ones dominate the loss.

The chart makes the shape of the problem concrete. Because the conditions compose multiplicatively, partial discipline buys very little: a pipeline that pins its dependencies and its container but derives its seeds from the wall clock reproduces almost nothing, and a pipeline that does everything except order its reductions reproduces its geometry and not its statistics — which is the worse of the two failures, because it looks like success.

Prerequisites & Toolchain

Nothing here needs a library you do not already have; what it needs is a place to put the record.

# requirements.txt — pin the majors, let patch float only if you also record what resolved
geopandas==0.14.4
shapely==2.0.4
pyproj==3.6.1
numpy==1.26.4

Two environment facts matter beyond the package list. PYTHONHASHSEED must be set to a fixed value, because Python’s string hashing is randomised per process by default and any code path that iterates a set or a dict of string keys inherits that randomisation. And the thread count of the numerical libraries must be pinned, because some linear-algebra and compression paths produce different results at different thread counts:

bash
export PYTHONHASHSEED=0
export OMP_NUM_THREADS=4
export OPENBLAS_NUM_THREADS=4
export MKL_NUM_THREADS=4

Recording these is as important as setting them. A run whose manifest does not say what the thread count was cannot be reproduced by somebody who does not already know.

Core Concept: The Derivation Tree

The central pattern is a tree. One root seed is recorded in the manifest; every stream any stage or partition uses is derived from it by hashing the root together with a stable name for that consumer. Nothing draws from a shared mutable generator, and nothing derives its stream from a counter.

python
import hashlib
import numpy as np


def derive(root: int, *parts: object) -> np.random.Generator:
    """Derive an independent generator from the root seed and a stable path.

    The path components must be properties of the *thing* being generated — a stage
    name, a region key, a tile's row and column — and never properties of the
    schedule, such as a worker id or a position in a work queue.
    """
    key = "/".join(str(p) for p in parts).encode()
    digest = hashlib.blake2b(key, digest_size=8, key=root.to_bytes(8, "big")).digest()
    return np.random.default_rng(int.from_bytes(digest, "big"))


root_seed = 20260811
sampler = derive(root_seed, "stage", "point-sampler")
tile_rng = derive(root_seed, "stage", "point-sampler", "tile", 12, 47)

The distinction in that docstring is the whole idea, and it is the one most implementations get wrong. Deriving a per-worker seed from a worker index gives every worker an independent stream, which looks correct and is not: re-running the job with a different number of workers, or with the same number in a different order, assigns different streams to the same work. Deriving from the tile’s own row and column gives the same tile the same stream forever, regardless of which worker picks it up, how many workers there are, or what order the scheduler dispatches in.

Three seed-derivation strategies against four re-run scenarios Rows are derivation strategies: drawing from one shared generator, deriving from the worker index, and deriving from the partition's own spatial key. Columns are re-runs: the identical run repeated on the same machine, the same run with a different number of workers, the same run with a different dispatch order, and a single partition regenerated on its own. A shared generator fails every column except the first, and it fails that one too as soon as the scheduler is free to interleave. Deriving from the worker index passes the repeated run and the reordered dispatch, because the indices are the same set either way, but fails as soon as the worker count changes or a partition is regenerated alone, since there is then exactly one worker and every partition would receive the first stream. Deriving from the partition's own spatial key — its row and column in the anchored grid — passes all four, because the key is a property of the partition rather than of the schedule. A closing note records the consequence for debugging: only the third strategy makes a failed reproduction bisectable, since only it lets one partition be regenerated and compared without re-running everything. Derive from what the partition is, not from when it ran Derivation same run different worker count different order one partition alone one shared generator ✗ ✗ ✗ ✗ from the worker index ✓ ✗ ✓ ✗ from the partition's spatial key ✓ ✓ ✓ ✓ Only the third strategy makes a failed reproduction bisectable: it is the only one under which a single partition can be regenerated and compared without re-running the whole job, which is the difference between an afternoon and a week when a release does not reproduce. A worker index is a property of the schedule. A row and column is a property of the work.
Three derivation strategies against the four re-runs that matter — only a content-addressed path survives all of them.

A second property falls out of the tree for free: partition-level reproducibility. Because a tile’s stream depends only on the root and the tile’s identity, a single tile can be regenerated in isolation and compared against the released one. That turns “the release does not reproduce” from a whole-pipeline investigation into a bisection over tiles, which is the difference between an afternoon and a week.

Step-by-Step Implementation

Step 1 — Resolve and freeze the environment before generating anything

Resolution happens once, at the top of the run, and its output is an input to everything below.

python
import json
import subprocess
import sys


def resolve_environment() -> dict:
    """Capture what actually resolved, not what was requested."""
    freeze = subprocess.run(
        [sys.executable, "-m", "pip", "freeze", "--disable-pip-version-check"],
        capture_output=True, text=True, check=True,
    ).stdout.splitlines()
    import pyproj
    import shapely
    return {
        "python": sys.version.split()[0],
        "packages": sorted(line for line in freeze if line and not line.startswith("-e ")),
        "proj": pyproj.proj_version_str,
        "geos": shapely.geos_version_string,
        "hash_seed": os.environ.get("PYTHONHASHSEED"),
        "threads": {k: os.environ.get(k) for k in
                    ("OMP_NUM_THREADS", "OPENBLAS_NUM_THREADS", "MKL_NUM_THREADS")},
    }

pyproj.proj_version_str and shapely.geos_version_string are worth calling out: the Python package version does not determine the PROJ or GEOS version, and those are the libraries whose behaviour actually changes the geometry. A wheel built against a different PROJ produces different datum-shift results from the same code.

Step 2 — Content-address every input asset

python
from pathlib import Path


def asset_digest(path: Path, chunk: int = 1 << 20) -> str:
    h = hashlib.blake2b(digest_size=16)
    with path.open("rb") as fh:
        while block := fh.read(chunk):
            h.update(block)
    return h.hexdigest()


inputs = {
    str(p.relative_to(asset_root)): asset_digest(p)
    for p in sorted(asset_root.rglob("*")) if p.is_file()
}

Sorting matters. An unsorted rglob yields filesystem order, which differs between machines and produces a different manifest for identical inputs — a manifest that changes when nothing changed is a manifest nobody trusts, and an untrusted manifest is quickly ignored.

Step 3 — Make every reduction ordered

python
def merge_partitions(results: dict[tuple[int, int], dict]) -> dict:
    """Combine per-tile results in a fixed order, so the aggregate is deterministic.

    `results` is keyed by tile identity, never by completion order.
    """
    total_features = 0
    area_sum = 0.0
    for key in sorted(results):                 # the fixed order lives here
        total_features += results[key]["features"]
        area_sum += results[key]["area"]
    return {"features": total_features, "area": area_sum}

The sort is the entire fix, and it costs nothing. What it buys is that the aggregate statistics — the ones a validation gate compares against a tolerance — no longer depend on which worker finished first.

Step 4 — Write the manifest as the run’s last act, and hash the output

python
def write_manifest(out_dir: Path, root_seed: int, env: dict, inputs: dict,
                   artifact: Path) -> Path:
    manifest = {
        "schema": "spatial-run/1",
        "root_seed": root_seed,
        "derivation": "blake2b(key=root, path='/'.join(parts))",
        "environment": env,
        "inputs": inputs,
        "artifact": {"path": artifact.name, "sha": asset_digest(artifact)},
    }
    path = out_dir / "manifest.json"
    # sort_keys so two identical runs produce byte-identical manifests
    path.write_text(json.dumps(manifest, indent=2, sort_keys=True))
    return path

Note what is not in the manifest: a timestamp, a hostname, a run id. Those belong in the run log, not in the reproducibility record, because including them means two identical runs produce different manifests and the manifest stops being comparable.

Manifest fields versus run-log fields, and the question each one answers The left column lists the manifest fields. The root seed and the derivation rule answer how every random stream in the run was produced; without them the run cannot be repeated at all. The resolved package list answers which code ran, as distinct from which code was requested. The native library versions — PROJ and GEOS — answer which geometry engine ran, and they are the fields most often omitted, because the Python package version does not determine them and a wheel built against a different PROJ produces different datum shifts. The container digest answers which image ran, as distinct from which tag was asked for. The input digests answer which bytes went in. The hash seed and thread counts answer two environment settings that change results silently and appear in no package list. The artifact hash answers what came out, and it is what a reproduction is compared against. The right column lists the run-log fields — timestamp, hostname, run identifier, wall-clock duration, worker count — each useful and none of them in the manifest, because including any of them means two identical runs produce different manifests and the manifest stops being comparable. A footer states the test: if a field would differ between two runs that produced identical output, it belongs in the log. If it differs between two runs that produced identical output, it is a log field MANIFEST — the reproducibility record root_seed how every stream was produced without it: the run cannot be repeated derivation rule how the root became per-stage streams without it: partitions cannot be isolated resolved packages which code ran, not which was asked for without it: a patch bump goes unnoticed PROJ + GEOS versions which geometry engine ran without it: datum shifts differ silently container digest which image ran, not which tag without it: the base changes underneath you input digests which bytes went in without it: 'the current version' is not an identifier hash seed + thread counts two settings in no package list without it: results change with no diff artifact sha what came out without it: there is nothing to compare against RUN LOG — useful, and not in the manifest timestamp when it ran hostname where it ran run id which invocation duration how long it took worker count how it was scheduled log level how loudly The test If a field would differ between two runs that produced identical output, it belongs in the log. A manifest that changes when nothing changed is a manifest nobody trusts, and an untrusted manifest is quickly ignored.
What belongs in the reproducibility record, what belongs in the run log, and the specific question each field is the only one able to answer.

Validation & Testing

The test for reproducibility is a re-run, and it belongs in CI rather than in somebody’s memory.

python
def test_run_is_reproducible(tmp_path):
    """Generate twice from the same manifest and compare artifact hashes."""
    manifest = json.loads(Path("fixtures/manifest.json").read_text())
    first = generate(manifest, tmp_path / "a")
    second = generate(manifest, tmp_path / "b")
    assert asset_digest(first) == asset_digest(second), (
        "identical manifests produced different artifacts"
    )


def test_single_tile_reproduces_in_isolation(tmp_path):
    """A tile regenerated alone must match the same tile from the full run."""
    manifest = json.loads(Path("fixtures/manifest.json").read_text())
    full = generate(manifest, tmp_path / "full")
    one = generate_tile(manifest, row=12, col=47, out=tmp_path / "one")
    assert extract_tile(full, 12, 47) == one.read_bytes()

The second test is the one that catches derivation mistakes, and it is the one teams skip. A pipeline can pass the first test — two full runs agreeing — while deriving its seeds from a worker index, provided both runs happen to use the same worker count. The isolation test fails immediately in that case, because a single-tile run has one worker.

A third check is worth running weekly rather than per-commit, because it is slow and it catches something the others cannot:

python
def test_reproduces_in_a_clean_container():
    """Re-run the pinned digest from a cold cache and compare."""
    ref = json.loads(Path("fixtures/manifest.json").read_text())
    out = docker_run(ref["environment"]["image_digest"], "generate", "--manifest", "-")
    assert out["artifact"]["sha"] == ref["artifact"]["sha"]

Where the Discipline Pays Back

The argument for this work is usually made as insurance, and insurance is a weak argument for engineering effort because the payout is hypothetical. Three of the returns are not hypothetical at all, and they arrive long before any audit does.

The first is debugging speed. A partition that can be regenerated in isolation turns “the release is wrong somewhere” into a bisection over partitions, and a bisection over ten thousand partitions is fourteen comparisons. Teams without partition-level reproducibility do not perform that bisection more slowly; they perform a different, worse investigation, in which the whole pipeline is re-run with print statements added.

The second is safe refactoring. A generator whose output is byte-reproducible has a free regression test: run the previous release’s manifest through the new code and compare hashes. A change that was meant to be a pure refactor and was not is caught in one command rather than by a consumer three weeks later. Without reproducibility that test does not exist, and the practical consequence is that generators calcify — nobody wants to touch code whose behaviour cannot be compared before and after.

The third is honest incident response. When a downstream team reports that something changed, the first question is whether the data changed or their pipeline did. With reproducible releases that question is settled by comparing two hashes. Without them it becomes a negotiation, and negotiations between teams about which of them broke something are expensive in a way that has nothing to do with engineering.

Performance & Scale Considerations

The derivation tree costs a hash per partition, which is nothing next to generating the partition. The two real costs lie elsewhere.

The first is that a fixed reduction order rules out the cheapest parallel reduction, which combines results as they arrive. In practice this matters far less than it sounds: the reduction is over per-partition summaries, not over the data, so it is thousands of small values rather than millions of large ones, and the sort is invisible in the profile.

The second is that content-addressing every input costs a full read of every asset. For a large raster collection that is real time. Two mitigations work: hash at ingest and store the digest alongside the asset, so the pipeline reads a recorded value rather than recomputing it; and for assets that are themselves produced by a pipeline, take the digest from their manifest rather than the file. Both preserve the property that matters, which is that the digest identifies the bytes rather than the path.

Failure Modes & Troubleshooting

  • The run reproduces on one machine and not another. Almost always an unpinned native library. Compare proj_version_str and geos_version_string between the two environments before looking at anything else; the Python package versions will usually match while the native ones do not.
  • The per-tile output matches and the aggregate does not. An unordered reduction. Sort the partition keys before combining, and check for a set or dict iteration in the merge path.
  • The run reproduces today and not next week. A floating tag or an unpinned patch version. Pin the container by digest and record the resolved package list rather than the requirement list.
  • A single tile does not reproduce in isolation. The seed derivation is keyed on something schedule-dependent — a worker index, an enumeration counter, a chunk position in a queue. Re-key it on the tile’s own coordinates.
  • The manifest differs between two identical runs. Something non-deterministic is being recorded: a timestamp, a hostname, an unsorted file listing. Move it to the run log.
  • Everything is pinned and output still differs. Check PYTHONHASHSEED and the thread-count variables. Both change results silently and neither shows up in a package list.

Frequently Asked Questions

Is a single seed at the top of the script enough?

Only for a single-threaded, single-stage script with no parallelism, which is not what a generation pipeline is. As soon as two stages or two workers draw from the same generator, the values each receives depend on the order they ran in, and that order is not part of your program. Derive per-stage and per-partition streams from the root instead.

Should the seed be random or fixed?

Random at the start of a new release, fixed thereafter. Drawing the root from the operating system’s entropy source means successive releases are genuinely independent samples rather than the same sample; recording it immediately means that release can be reproduced. What must never happen is deriving the root from the clock and not recording it, which produces a release that is independent and unreproducible at once.

Do I need this if the output is not bit-exact anyway?

Bit-exactness is the only version of reproducibility that can be checked cheaply. “Statistically equivalent” requires choosing a statistic, a tolerance and a test, all of which are arguable, and none of which will settle a dispute about whether two releases differ. Aim for byte-identical, and where a genuinely non-deterministic component makes that impossible — some GPU kernels, for instance — isolate that component and record its output as an input to the rest.

How does this interact with the privacy budget?

Regenerating a release from its manifest does not spend additional budget, because it produces the same artifact rather than a new observation of the population — but only if it genuinely reproduces. A “regeneration” that draws fresh noise is a second release and must be accounted as one, which is another reason the reproduction test matters. See diagnosing epsilon budget exhaustion across releases.