Artifact Versioning & Lineage for Synthetic Spatial Releases

A synthetic spatial dataset is not a file. It is a version of a claim about a population, produced by a specific generator from specific inputs under a specific privacy budget, consumed by systems that will still be running long after that version has been superseded. This page is part of Synthetic Spatial Data Architecture & Fundamentals, and it covers the registry that makes those relationships queryable — the one written at promotion time, as distinct from the seed and run registry written at generation time.

The distinction is not pedantry. The seed registry answers how was this produced. The artifact registry answers what may consume this, and for how long. A platform that conflates them cannot tell a run that was produced but never promoted — exactly the run you want to inspect after a validation failure — from one that shipped.

Problem Framing: Three Questions That Become Unanswerable

Every team building synthetic spatial data eventually faces the same three questions, usually in the same order, and usually for the first time during an incident.

Which release is this model actually trained on? A downstream team reports that their model’s behaviour changed. They believe they are using “the latest” synthetic release. Without a registry, establishing what they are actually using means asking them, which means asking whoever configured their pipeline, which means reading a config file that names a path rather than a version. Paths are mutable; a path is not an identity.

What changed between these two releases? Two releases behave differently and both passed every gate. The gates verified that each release was individually acceptable, which is a different property from the two being comparable. Answering the question requires knowing what inputs, what generator version and what parameters differed — and none of that is recoverable from the artifacts themselves.

Who consumes the release we just found a defect in? This is the one that turns a quality problem into an operational one. A defect is discovered in a release that shipped two months ago. Notifying the affected consumers requires a list of them, and that list exists only if consumption was recorded at the time.

Time to answer three lineage questions, with and without a promotion-time registry Three questions group the horizontal axis. The vertical axis is time to an answer, on a logarithmic scale from seconds to weeks, because the two conditions differ by orders of magnitude rather than by percentages. With a registry, each question is a query and the answer arrives in seconds. Without one, the first question — which release a given consumer is actually using — requires tracing a path through somebody's configuration and takes hours. The second — what changed between two releases — requires reconstructing two environments and takes days, and often ends inconclusively because the generator version was never recorded. The third — who consumes a release found to be defective — cannot be answered at all without consumption records; the bar is drawn at the ceiling and marked as an estimate of the broadcast-and-wait approach that replaces it. The note underneath states the property that unites them: none of the three is a hard engineering problem, and all three are impossible to solve retroactively, which is why they are worth solving before they are first asked. None of these is hard. All three are impossible to solve retroactively. 1 s 10 s 2 min 17 min 3 h 1 d 12 d time to an answer (log scale) seconds hours which release is this consumer using? seconds 3 days what changed between these two releases? seconds 10 days who consumes the release we just found a defect in? — and often not at all: this is broadcast-and-wait registry written at promotion time reconstructed afterwards The third question is the one that turns a quality problem into an operational one, and it is the only one of the three that cannot be answered at all without a record made at the time.
The three questions, and the difference in time-to-answer between a platform that recorded lineage at promotion time and one reconstructing it after the fact.

None of the three is a hard engineering problem. All three are impossible to solve retroactively, which is why they are worth solving before they are asked.

Prerequisites & Toolchain

The registry does not need a specialist system. A table, an object store, and the discipline to write to both at the same moment are sufficient — and a simpler store that is always written is worth more than a sophisticated one that is written when somebody remembers.

# What the registry needs to reference, and where each lives
artifact bytes        → object store, keyed by content hash
manifest              → object store, beside the artifact
registry rows         → any transactional store (Postgres, SQLite, a versioned table)
consumption events    → an append-only log, written by consumers

The one hard requirement is that the artifact is addressed by the hash of its bytes rather than by a path. Everything else in this page follows from that: a content-addressed artifact cannot be silently replaced, two releases that are byte-identical are recognisably so, and a reference to a release is a reference to specific bytes rather than to whatever currently sits at a location.

Core Concept: Version, Identity and Validity Are Three Different Things

The commonest design mistake here is a single version number carrying three unrelated meanings.

Identity is the content hash. It answers “are these the same bytes?” and nothing else. It is not human-readable, it does not sort, and it does not tell you which of two releases is newer — and it should not, because those are different questions.

Version is a human-facing label with an ordering: 2026.2.0. It answers “which is newer” and “is this a breaking change”. Its components should mean something the consumer can act on — a major bump means the schema or the semantics changed, a minor bump means new data under the same contract, a patch means a defect was corrected without a semantic change.

Validity is a time window plus a status. It answers “may I use this now”. A release can be current, superseded, deprecated or withdrawn, and those are not the same: superseded means a newer one exists and this one is still correct, whereas withdrawn means this one is wrong and must not be used.

Identity, version and validity as three separate fields, and what collapsing them costs Three cards side by side. Identity is the content hash: it answers whether two releases are the same bytes, it must not be used for ordering because a hash does not sort meaningfully, and without it a release can be silently replaced at the same path. Version is the semantic label: it answers which release is newer and whether the change is breaking, it must not be used as an identity because two builds can carry the same version and differ, and without it a consumer has no basis on which to decide whether upgrading is safe. Validity is a status and a time window: it answers whether a release may be used right now, it must not be inferred from the version ordering, and without it there is no way to say stop. Beneath the three cards, a banner names the operation that only the separation makes possible: withdrawing a release without publishing a replacement. A platform whose only signal is that a newer version exists cannot tell a consumer to stop using something and not upgrade, because there is nothing correct yet to upgrade to — and that is precisely the message a serious defect requires. Three fields, three questions — and one message only the third can send Identity the content hash ANSWERS are these the same bytes? NOT FOR not for ordering — a hash does not sort WITHOUT IT without it, a release is silently replaceable at the same path Version the semantic label ANSWERS which is newer, and is it breaking? NOT FOR not as an identity — two builds can share a version WITHOUT IT without it, a consumer cannot decide whether upgrading is safe Validity status + time window ANSWERS may I use this right now? NOT FOR not inferred from the version ordering WITHOUT IT without it, there is no way to say stop The operation only the separation makes possible: withdraw without replacing. A consumer told only that a newer version exists will upgrade and carry on — the wrong move when the correct action is to stop. Superseded means a newer release exists and this one is still correct. Withdrawn means this one is wrong. Collapsing them into a version comparison makes the two indistinguishable to every consumer.
Three fields answering three questions — collapsing them into one version number is what makes a withdrawal indistinguishable from an upgrade.

Keeping them separate makes an important operation expressible: withdrawing a release without publishing a replacement. A platform whose only signal is “a newer version exists” cannot say “stop using this and do not upgrade, because we do not yet have anything correct to give you”. That message is exactly what a serious defect requires.

Step-by-Step Implementation

Step 1 — Promote by writing a row, not by moving a file

python
import json
import hashlib
from dataclasses import dataclass, asdict
from datetime import date


@dataclass(frozen=True)
class Release:
    artifact_sha: str          # identity — the hash of the bytes
    version: str               # ordering — semantic, human-facing
    status: str                # current | superseded | deprecated | withdrawn
    valid_from: str
    valid_until: str | None
    generator_version: str
    manifest_sha: str
    input_shas: dict[str, str]
    privacy_ledger_row: str    # the ε spend this release was debited against
    supersedes: str | None     # the artifact_sha this replaces, if any


def promote(release: Release, registry) -> None:
    """Promotion is a registry write. The bytes were already in the store."""
    registry.insert(asdict(release))
    if release.supersedes:
        registry.update(release.supersedes, status="superseded")

Two properties matter here. Promotion does not move or copy the artifact — the bytes went into the content-addressed store when they were produced, and promotion only records that they may now be consumed. And superseding is an explicit edge rather than an implication from the version ordering, so the chain of what replaced what survives a version-numbering change.

Step 2 — Record lineage as a graph, not as a string

python
def lineage(registry, artifact_sha: str, depth: int = 6) -> dict:
    """Walk backwards through inputs that are themselves registered artifacts."""
    row = registry.get(artifact_sha)
    node = {
        "artifact": artifact_sha,
        "version": row["version"],
        "generator": row["generator_version"],
        "inputs": [],
    }
    if depth:
        for name, sha in sorted(row["input_shas"].items()):
            upstream = registry.get(sha)
            node["inputs"].append(
                lineage(registry, sha, depth - 1) if upstream
                else {"external": name, "sha": sha}
            )
    return node

The recursion terminates on inputs that are not themselves registered — a boundary file from a national mapping agency, say — and records them as external with their digest. That boundary is worth being explicit about, because it is where the platform’s guarantee ends: everything inside the graph can be regenerated, and everything at the leaves has to be retained.

Step 3 — Make consumption an event

python
def resolve(registry, name: str, at: str | None = None) -> Release:
    """Consumers resolve a name to a specific artifact, and the resolution is logged."""
    row = registry.current(name, at=at)
    if row["status"] == "withdrawn":
        raise ValueError(f"{name} {row['version']} was withdrawn: {row['withdrawal_reason']}")
    registry.log_consumption(artifact_sha=row["artifact_sha"], consumer=caller_identity())
    return Release(**row)

Making resolution the only supported way to obtain a release is what turns the third question — who consumes this — from an archaeology exercise into a query. Consumers who copy a path out of a config file and read the object store directly are invisible, so the resolution API has to be more convenient than the alternative, not merely available.

Step 4 — Give every release an expiry, and mean it

python
DEFAULT_VALIDITY_DAYS = 180


def with_expiry(release: Release, days: int = DEFAULT_VALIDITY_DAYS) -> Release:
    return replace(release, valid_until=(date.fromisoformat(release.valid_from)
                                         + timedelta(days=days)).isoformat())

An expiry is not a deletion. It is a statement that the population this release describes has moved on far enough that the release should no longer be assumed representative, and it forces a decision — renew, regenerate, or accept the staleness explicitly — rather than allowing a four-year-old synthetic dataset to quietly remain in a training pipeline because nothing ever said to stop.

Step 5 — Make the registry the thing that answers, not a place to look things up

A registry that has to be read by a human is a registry that goes stale, because the effort of keeping it accurate is paid by somebody who is not the one benefiting. The registries that survive are the ones that sit on the critical path: nothing is promoted except by writing a row, and nothing is consumed except by resolving a name. Both of those are cheap, and both of them fail loudly, which is what keeps the record true.

That principle has a corollary worth stating, because it is the commonest way these systems decay. Any convenience that lets a consumer skip resolution — a stable “latest” path in the object store, a symlink, a shared mount — removes that consumer from the record without removing them from the dependency graph. They are still affected by a withdrawal and they are no longer notifiable. If such a path exists because resolution was inconvenient, the fix is to make resolution convenient rather than to write a policy forbidding the shortcut.

python
def latest(name: str) -> Path:
    """The convenience consumers actually want — routed through the registry."""
    release = resolve(registry, name)          # logs consumption, refuses withdrawn releases
    return store.local_path(release.artifact_sha)

Ten lines of that kind remove the incentive to bypass the system, which is worth more than any amount of documentation about why bypassing it is unwise.

Validation & Testing

python
def test_every_promoted_release_is_resolvable():
    """No registry row may reference bytes the store does not have."""
    for row in registry.all():
        assert store.exists(row["artifact_sha"]), row["version"]
        assert store.exists(row["manifest_sha"]), row["version"]


def test_supersession_chain_is_acyclic():
    for row in registry.all():
        seen, cur = set(), row["supersedes"]
        while cur:
            assert cur not in seen, f"cycle at {cur}"
            seen.add(cur)
            cur = registry.get(cur)["supersedes"]


def test_no_current_release_is_expired(today):
    stale = [r for r in registry.where(status="current")
             if r["valid_until"] and r["valid_until"] < today]
    assert not stale, f"{len(stale)} expired releases still marked current"


def test_withdrawn_releases_are_not_resolvable():
    for row in registry.where(status="withdrawn"):
        with pytest.raises(ValueError):
            resolve(registry, row["name"], at=row["valid_from"])

The third of these is the one that catches drift in practice. Expiry only works if something enforces it, and the enforcement is a scheduled check rather than a hope.

Performance & Scale Considerations

The registry is small — one row per release, not per feature — so nothing here is a throughput problem. Two operational costs are worth planning for.

The first is retention of the bytes. Content-addressing means two releases that share an input share its storage, which helps, but a platform that retains every release forever will accumulate. The rule that works is to retain the artifact for as long as any consumer’s recorded consumption event references it, plus a margin, and to retain the manifest and registry row indefinitely — the manifest is kilobytes and it is what makes a withdrawn release explicable years later.

The second is the lineage walk. Recursion over a deep graph with a database round trip per node is slow enough to discourage use, and a lineage query nobody runs is a lineage record nobody trusts. Materialise the closure — the transitive set of upstream artifacts — as a table maintained on promotion, and the query becomes a single lookup.

Retention classes for a release, by size, rule and what their loss costs Four rows, ordered by size. The artifact bytes are by far the largest, typically gigabytes, and they are the only class where retention is genuinely expensive; the rule is to keep them for as long as any recorded consumption event references them, plus a margin, and once they are gone the release can no longer be inspected directly — though it can still be regenerated, provided the manifest survives. The manifest is kilobytes and should be retained indefinitely, because it is what makes a withdrawn release explicable years later and what allows the bytes to be regenerated at all. The registry row is bytes and should also be retained indefinitely; without it a hash found in a consumer's configuration cannot be connected to anything. The consumption log grows steadily and can be compacted to distinct consumer-artifact pairs rather than every event, which keeps the question of who consumes a release answerable at a fraction of the volume. A closing note makes the asymmetry explicit: the cheapest classes to retain are the ones whose loss is unrecoverable, and the expensive one is the only one that can be rebuilt. The cheapest things to keep are the ones whose loss cannot be undone artifact bytes GB RETAIN as long as any consumption event references them, plus a margin ONCE GONE cannot be inspected directly — but can be regenerated from the manifest manifest kB RETAIN indefinitely ONCE GONE the release becomes unregenerable and inexplicable registry row bytes RETAIN indefinitely ONCE GONE a hash in a config cannot be connected to anything consumption log MB, compactable RETAIN compact to distinct (consumer, artifact) pairs ONCE GONE 'who consumes this' becomes unanswerable again Only the first row is expensive to keep, and it is the only one that can be rebuilt. The three cheap rows are the ones whose loss is permanent — which is the opposite of how retention policies are usually written.
What to keep and for how long: the bytes are the expensive part and the cheapest thing to retain is the one that makes a withdrawn release explicable years later.

Failure Modes & Troubleshooting

  • Two releases have the same version and different hashes. The version was assigned before promotion, or assigned by hand. Assign it at promotion, in the same transaction as the row.
  • A consumer is using a withdrawn release. They bypassed the resolution API. The fix is not a policy but an ergonomics one: find out why the direct path was easier and close that gap.
  • The lineage graph has a cycle. An artifact was registered as an input to something it descends from, usually because a “corrected” release was fed back in under the same name. Supersession is an edge; correction produces a new artifact rather than mutating an old one.
  • The registry references bytes the store no longer has. Retention ran ahead of consumption. Retention must key on consumption events, not on age alone.
  • Nobody can say what changed between two releases. The generator version was not recorded, or it was recorded as a branch name rather than a commit. Record the commit and the resolved dependency set, both of which the run manifest already contains.

The One Field Teams Add Last

There is a field that almost never appears in a first version of a registry and almost always appears in the second: the reason. Not the reason a release exists, which is usually obvious, but the reason for each status transition — why this release superseded that one, why this one was deprecated, why this one was withdrawn.

It is easy to see why it gets left out. At the moment of a transition the reason is completely obvious to whoever is performing it, so recording it feels like ceremony. Six months later it is obvious to nobody, and the questions that arrive are exactly the ones a reason field answers: was this release withdrawn because of a defect in the data or because of a defect in the contract it was published against? Was that one deprecated because it went stale, or because the source population changed in a way that makes the whole series suspect?

Those two pairs have very different consequences for a consumer, and a status alone cannot distinguish them. A free-text reason plus a small controlled vocabulary — superseded_by_schedule, defect_in_data, defect_in_contract, source_changed, budget_exhausted — costs one column and turns the registry from a record of what happened into a record of why, which is the version somebody can actually act on.

Frequently Asked Questions

Is semantic versioning the right scheme for a dataset?

The mechanics transfer; the meanings need restating. A major bump should mean a consumer’s code may break — the schema changed, a field’s units changed, the spatial extent moved. A minor bump should mean new data under the same contract. A patch should mean a defect was corrected with no semantic change. What does not transfer is the assumption that a consumer can safely take the latest minor: a synthetic dataset’s statistical properties can shift within a minor version, so the contract has to say which properties are guaranteed stable and the gates have to enforce it.

Should the registry store the privacy budget?

It should reference it, not duplicate it. The privacy ledger is its own append-only record keyed on the source population, and a release row carries the identifier of the ledger entry it was debited against. Duplicating the epsilon into the registry creates two numbers that can disagree, and when they do, neither is trusted. See diagnosing epsilon budget exhaustion across releases.

How do I handle a release that is wrong but widely consumed?

Withdraw it rather than superseding it, publish the reason in the registry row, and use the consumption log to notify the specific consumers rather than announcing broadly. Withdrawal without a replacement is the case the status field exists for: a consumer told only that a newer version exists will upgrade and carry on, which is the wrong action when the correct one is to stop.