Running Spatial Validation on a Pull-Request Budget

The full validation suite takes forty minutes. Nobody will wait forty minutes on a pull request, so the suite runs nightly, so defects are found a day after they are introduced by somebody who has moved on to something else.

Part of CI/CD Integration for Spatial Data: this page is about the split — what runs on every commit, what runs before a release, and how to make the first of those genuinely useful rather than a token.

Root Cause: Validation Cost Is Concentrated, and So Is Its Value

Two distributions matter, and they are not the same shape.

Cost is heavily concentrated. In a typical spatial suite, the adversarial privacy checks and the full-extent generation account for most of the wall clock, and the manifest and structural checks account for almost none. That is the distribution most teams know about.

Value on a pull request is concentrated too, and in the opposite place. A pull request changes code; the defects it introduces are overwhelmingly structural and contractual — a CRS assertion removed, a schema field renamed, a geometry repair that now produces slivers. The expensive statistical and adversarial checks mostly detect data drift, which a code change rarely causes and which a nightly run catches perfectly well.

Runtime share and pull-request defect share across seven gate groups Seven gate groups run along the horizontal axis, ordered from cheapest to most expensive: manifest checks, schema checks, geometry validity, integrity scans, distributional metrics, full-extent generation, and adversarial privacy checks. Each carries two bars. The first is the group's share of the suite's total runtime, which rises steeply from left to right — the last two groups together account for the large majority of the wall clock. The second is the group's share of the defects that a code change actually introduces, measured against a history of pull requests, and it falls from left to right: a code change overwhelmingly breaks manifests, schemas and geometry handling, and only rarely produces the statistical or adversarial failures that dominate the runtime. The two distributions are near mirror images. The note underneath states the consequence that follows: a pull-request suite built from the cheap groups plus a small sample of the expensive ones catches most of what a code change breaks in a small fraction of the time, and the expensive groups are better spent on the nightly run where they catch data drift instead. Cost rises left to right; pull-request value falls 0% 10% 20% 30% 40% 50% share (%) 1 2 5 7 14 27 44 26 21 24 12 9 5 3 manifest schema geometry integrity distributional full-extent generation adversarial privacy share of suite runtime share of defects a code change introduces The two distributions are near mirror images, which is the whole opportunity: the cheap groups plus a small sample of the expensive ones catch most of what a code change breaks, and the expensive groups earn their place on the nightly run catching data drift instead.
Cost and pull-request value across the gate set: the cheapest gates catch most of what a code change actually breaks.

That mismatch is the whole opportunity. A pull-request suite made of the cheap gates plus a small sample of the expensive ones catches the great majority of what a code change breaks, in a couple of minutes.

Prerequisite Check: Measure the Suite Before Splitting It

python
import time
from contextlib import contextmanager


@contextmanager
def timed(name: str, into: dict):
    t = time.perf_counter()
    yield
    into[name] = time.perf_counter() - t


def profile_suite(gates: list, artifact) -> dict:
    timings = {}
    for gate in gates:
        with timed(gate.name, timings):
            gate.run(artifact)
    total = sum(timings.values())
    return {"total_s": total,
            "share": {k: v / total for k, v in sorted(timings.items(),
                                                      key=lambda kv: -kv[1])}}

Run this before deciding anything. The intuition about which gate is slow is wrong more often than not — a check that feels heavy because it is conceptually complicated is frequently cheaper than a simple one that happens to read the whole artifact twice.

Fix: Three Tiers, With Sampling in the Middle

Tier one — the manifest, on every commit, in under a second

python
MANIFEST_GATES = ["crs_declared", "envelope_declared", "schema_declared",
                  "seed_present", "contract_version_pinned", "tolerances_declared"]

These read the manifest and nothing else. They catch a removed assertion, a renamed field and an unpinned contract — the exact failures a code change produces — and they cost nothing, so they can run before anything else and fail fast.

Tier two — a deterministic sample, on every commit, in a couple of minutes

The expensive gates become affordable when run on a fixed, small, deterministically chosen extent:

python
import hashlib


def pr_sample(contract: dict, n_tiles: int = 4) -> list[tuple[int, int]]:
    """A small, fixed set of tiles — same on every run, chosen to be representative.

    Derived from the contract rather than the commit, so the sample does not move
    between pull requests and two runs are comparable.
    """
    tiles = contract["tile_index"]
    key = contract["contract_version"].encode()
    scored = sorted(tiles, key=lambda t: hashlib.blake2b(
        f"{t}".encode(), key=key, digest_size=8).digest())
    # one tile from each declared area type, so the sample is not all rural
    by_type: dict = {}
    for t in scored:
        by_type.setdefault(tiles[t]["area_type"], t)
    return sorted(by_type.values())[:n_tiles]

Two properties matter. The sample is fixed — derived from the contract, not from the commit or the clock — so two pull-request runs are comparable and a failure is reproducible. And it is stratified by area type, because an unstratified sample of four tiles is usually four rural tiles, and rural tiles pass everything.

Tier three — the full suite, before a release only

Everything else: the full extent, the adversarial privacy checks, the perceptual and duplicate scans over the whole artifact. These run on the release branch and on a schedule, and their job is to catch data drift rather than code defects.

Three validation tiers against a five-minute pull-request budget A horizontal time axis runs from zero to forty-five minutes on a logarithmic scale. Three bars sit above it. Tier one, the manifest gates, finishes in well under a second: it reads the manifest and nothing else, and it fails fast on a removed assertion or an unpinned contract, which is what a code change usually breaks. Tier two, the structural and distributional gates run on a fixed four-tile stratified sample plus a cached fixture, takes just over two minutes. Together they finish inside a dashed five-minute budget marker with room to spare. Tier three, the full suite over the whole extent including the adversarial checks, extends to forty minutes and is drawn past the budget line, annotated as belonging to the release branch and the nightly schedule. A second annotation notes what the fixture cache removes from tier two: without it, regenerating the sample extent would roughly triple that tier's time and push the pair past the budget. Two tiers inside the budget, one outside it on purpose tier 1 manifest gates 0.8 s reads the manifest, nothing else tier 2 sampled structural + distributional 2 min 4 stratified tiles, cached fixture tier 3 full suite, whole extent 40 min release branch and nightly only 5-minute pull-request budget 1 s 10 s 1 min 10 min 45 min Without the fixture cache, tier two roughly triples and the pair no longer fits — which makes the cache a budget decision rather than an optimisation.
The three tiers against a five-minute pull-request budget: tier one and a four-tile tier two fit inside it with room to spare.

Fix, Part Two: Cache the Fixture, Not the Result

The other half of the budget goes on producing something to validate. Regenerating a sample extent on every pull request is usually the single largest line item, and it is avoidable:

python
def fixture_key(contract: dict, generator_version: str, tiles: list) -> str:
    """A fixture is reusable while the contract, the generator and the tiles are unchanged."""
    h = hashlib.blake2b(digest_size=16)
    for part in (contract["contract_version"], generator_version, repr(sorted(tiles))):
        h.update(part.encode())
    return h.hexdigest()

Keying on the generator version means the fixture is invalidated exactly when the code that produces it changes — which is the correct behaviour, and is also why the cache hits on the many pull requests that change validation code, documentation or configuration rather than the generator itself.

Caching the result rather than the fixture is the tempting shortcut and it is wrong: it means a pull request that changes only a validation threshold will report the previous verdict.

Verification Step: Assert the Budget Itself

python
import pytest

PR_BUDGET_S = 300


def test_pr_suite_fits_the_budget(pr_gates, fixture):
    profile = profile_suite(pr_gates, fixture)
    assert profile["total_s"] < PR_BUDGET_S, (
        f"pull-request suite takes {profile['total_s']:.0f}s; "
        f"slowest: {list(profile['share'])[:3]}"
    )


def test_tier_one_fails_fast(pr_gates, broken_manifest):
    """A manifest defect must fail before any generation happens."""
    t = time.perf_counter()
    with pytest.raises(ValidationError):
        run_suite(pr_gates, broken_manifest)
    assert time.perf_counter() - t < 2.0


def test_sample_is_stable_across_runs(contract):
    assert pr_sample(contract) == pr_sample(contract)

Asserting the budget in the suite is what stops it drifting back. A suite with no budget test grows by thirty seconds a quarter and is back at forty minutes within two years, at which point somebody moves it to nightly again and the cycle repeats.

Edge Cases & Gotchas

The sample passes and the full extent fails. Expected occasionally, and it is not a failure of the approach — it is the tier-three run doing its job. What matters is whether the failure class is one the sample could have caught: if it is, the stratification needs another axis.

Defect detection against sample size, for uniform and stratified tile sampling The horizontal axis is the number of tiles in the pull-request sample, from two to twelve. The vertical axis is the probability that the sample contains at least one tile exhibiting a defect. The lower curve is uniform random sampling: because rural tiles are the great majority and rural tiles rarely exhibit the defects a code change introduces, a small uniform sample is usually four rural tiles and detects little. The upper curve is a sample stratified by area type, taking one tile from each declared type in turn: it reaches a substantially higher detection rate at every size, and at four tiles it detects roughly what a uniform sample of twelve does. A marker shows the four-tile comparison. The note underneath records why the gap is so large: the defect rate is highest in the area types that are rarest, so sampling in proportion to frequency systematically under-samples exactly where the defects are — and stratifying costs nothing beyond declaring the area type on each tile. The rarest area types are where the defects are 2 4 6 8 12 0% 25% 50% 75% 100% tiles in the pull-request sample samples containing at least one defect (%) stratified at 4 tiles: 71% uniform at 4 tiles: 33% stratified by area type uniform random 4,000 simulated samples per size, fixed seed; defect rates rise as area types get rarer. Sampling in proportion to frequency under-samples exactly where the defects are, and stratifying costs nothing beyond declaring the area type on each tile.
Measured over 4,000 simulated samples: a stratified sample of four tiles detects roughly what a uniform sample of twelve does.

A pull request that changes the contract. The fixture key changes, so the cache misses and the run is slow. That is correct: a contract change is exactly when the expensive validation is worth paying for.

Flaky expensive gates. A gate that fails intermittently on the full extent will fail intermittently on the sample too, and on a pull request that is far more damaging because it blocks somebody. Fix the flake or move the gate to tier three; a gate people learn to re-run is a gate that is no longer read.

Everything already fits. Then the split is not needed and adding it is complexity for nothing. Profile first; the answer is sometimes that the suite is fine and the perception of slowness came from a single pathological gate.

What Happens to the Gates That Moved

Splitting a suite moves gates out of the pull request, and it is worth being deliberate about where they land, because “not on the pull request” is not a destination.

The release branch gets the full suite, and it runs before promotion rather than after merge. That is the important detail: a gate that runs after merge finds defects in code that is already on the main branch, which means the fix is a second change rather than an amendment to the first.

The nightly schedule gets the gates whose job is detecting data drift rather than code defects — the distributional comparisons against a reference, the duplicate scans over the full artifact, the adversarial checks. Their failure mode is gradual, so daily is frequent enough, and running them on a schedule rather than on a trigger means their cost is predictable.

Nothing gets dropped. The temptation, once a gate is out of the pull request, is to notice that it has not failed in months and quietly remove it. The gates that have not failed in months are usually the ones holding a property everybody now takes for granted, which is exactly the property that will regress the moment the gate is gone.

One more piece is worth adding at the same time: a weekly report of which gates fired, in which tier, over the previous week. It costs nothing to produce and it is the only thing that makes the split reviewable — a tier-three gate that fires regularly is one that belongs in tier two, and a tier-two gate that has never fired is one worth questioning.