Setting Up Automated Fallbacks for Missing Spatial Attributes

When a synthetic batch arrives with null geometries, an undefined coordinate reference system, or absent attribute joins, an unguarded pipeline either crashes or — worse — silently imputes a value that no consumer can trace; this page shows how to replace ad-hoc imputation scripts with a deterministic, contract-governed fallback router that records every substitution and halts when missingness exceeds the agreed budget.

Part of Scoping Rules & Data Contracts: where the parent area defines the spatial envelope and the machine-readable interface agreement a synthetic pipeline must honour, this page resolves one concrete task — what a stage should do when an attribute the contract expects is missing, so that the substitution is reproducible, auditable, and itself subject to the same validation as primary-generation output.

Root Cause: Why Missing Attributes Are Silent Failures

Missing spatial attributes are not a single defect; they are four distinct gaps that surface at different stages. A generator emits an empty geometry (ST_IsEmpty returns true) when a sampler fails to place a feature. A stage drops the CRS tag, leaving coordinates that are “just numbers” with no datum. An attribute join misses, so a parcel arrives with geometry but no land-use class. Or a topological relationship — a road that should snap to a network node — is never formed. Each gap is invisible at the point of origin because the record still serializes as well-formed GeoJSON.

The temptation is to patch these with inline imputation: a fillna(0), a buffer(0), a hard-coded EPSG:4326. That introduces uncontrolled variance the moment it runs. A coordinate filled with 0 lands on Null Island; a default CRS reprojects nothing and corrupts everything downstream; a mean-filled land-use class flattens the spatial autocorrelation that the realism metrics and evaluation layer scores the batch against. Because the imputation is undocumented, the resulting drift is impossible to attribute to a cause weeks later.

The fix is to treat fallback as a first-class pipeline stage governed by the same CRS and schema contract that governs generation. The contract declares which attributes are mandatory, which are conditionally required, and which may be synthesized through a fallback chain — and, critically, the maximum fraction of a batch that may undergo fallback before the run is quarantined. A fallback that logs a confidence score and trips a CI gate surfaces a data-quality problem; one that silently fills a value hides it. The remainder of this page builds the former.

Contract-governed fallback router: Detect to Classify to Execute to Validate, then promote or quarantine A horizontal four-stage pipeline. The data contract sits on top and feeds mandatory/conditional attribute lists and the fallback budget downward into the Detect stage and the Validate stage. Detect runs a vectorized scan in parallel with generation, flagging null or empty geometry, undefined CRS, the Null Island sentinel, and missing mandatory attribute joins. Classify routes each gap to a strategy bucket using a versioned registry keyed on geometry type and the MCAR, MAR or MNAR missingness mechanism. Execute applies the selected strategy — centroid_buffer, network_snap, spatial_knn or crs_normalize — with a seed derived from the batch id, record hash and strategy version so it is idempotent, and emits an immutable audit record. Validate enforces topology validity, a two-sample Kolmogorov-Smirnov test and Moran's I against the contract tolerance band. A passing batch is promoted to a hash-chained ledger along the solid path; a batch whose fallback rate exceeds the budget or that produced an invalid geometry is routed to quarantine along the dashed accent path. Data contract mandatory / conditional attrs · fallback budget seed policy · tolerance band thresholds budget + tolerances 1 · Detect vectorized scan · parallel null / empty geometry undefined CRS · Null Island missing mandatory join 2 · Classify versioned registry lookup by geometry type MCAR / MAR / MNAR → strategy bucket 3 · Execute seed = hash(batch,rec,ver) centroid_buffer · net_snap spatial_knn · crs_normalize idempotent · audit record 4 · Validate topology · ST_IsValid KS two-sample test Moran's I check vs tolerance band Promote hash-chained ledger pass Quarantine fallback rate > budget or invalid geometry / drift fail

Minimal Reproducer: Confirming the Gap

Before wiring a router, reproduce the conditions it must catch so the fix is verifiable. The snippet below builds a small frame that exhibits all four gaps and shows that none of them raises on its own.

python
import geopandas as gpd
from shapely.geometry import Point, Polygon

gdf = gpd.GeoDataFrame(
    {
        "land_use": ["residential", None, "commercial"],   # missing attribute join
        "geometry": [
            Point(-122.42, 37.77),
            Polygon(),                                      # empty geometry
            Point(0, 0),                                    # Null Island sentinel
        ],
    },
    crs=None,                                               # undefined CRS
)

print("CRS defined     :", gdf.crs is not None)            # False
print("empty geometries:", gdf.geometry.is_empty.sum())    # 1
print("null attributes :", gdf["land_use"].isna().sum())   # 1
print("serializes ok   :", gdf.to_json()[:1] == "{")       # True — nothing raised

The frame serializes cleanly, which is exactly why the gaps must be caught by an explicit scan rather than an exception.

Fix: A Detect → Classify → Execute → Validate Router

The router is a stateless module with four sequential stages. Each fallback is deterministic — its randomness is seeded from the batch ID, the record hash, and the strategy version — so a re-run produces byte-identical output and the audit trail is reproducible across CI runs.

Stage 1 — Detect

Scan the batch with vectorized checks rather than row-by-row iteration. Detection runs in parallel with primary generation so it never blocks I/O.

python
import geopandas as gpd
import pandas as pd

MANDATORY = {"land_use"}          # from the data contract

def detect_gaps(gdf: gpd.GeoDataFrame) -> pd.DataFrame:
    return pd.DataFrame({
        "crs_undefined":   gdf.crs is None,
        "geom_missing":    gdf.geometry.is_empty | gdf.geometry.isna(),
        "null_island":     gdf.geometry.geom_type.eq("Point") & gdf.geometry.x.eq(0) & gdf.geometry.y.eq(0),
        "attr_missing":    gdf[list(MANDATORY)].isna().any(axis=1),
    })

Stage 2 — Classify

Route each gap to a strategy bucket keyed on geometry type, the missingness mechanism (MCAR, MAR, MNAR), and the downstream consumer. Classification is a lookup against a versioned registry, not branching logic scattered through the codebase.

python
def classify(gap_row, geom_type: str) -> str:
    if gap_row["crs_undefined"]:
        return "crs_normalize"
    if gap_row["geom_missing"]:
        return "centroid_buffer" if geom_type.endswith("Polygon") else "network_snap"
    if gap_row["attr_missing"]:
        return "spatial_knn"
    return "noop"

Stage 3 — Execute

Apply the selected strategy with a seed derived from immutable batch state. Execution must be idempotent so a pipeline retry never compounds an earlier substitution.

python
import hashlib
import numpy as np
import geopandas as gpd

def make_seed(batch_id: str, record_hash: str, strategy_version: str) -> int:
    digest = hashlib.sha256(f"{batch_id}:{record_hash}:{strategy_version}".encode()).hexdigest()
    return int(digest[:16], 16) % (2**32)

def centroid_buffer(parent: gpd.GeoSeries, max_radius_m: float) -> gpd.GeoSeries:
    # Replace a missing polygon with the parent unit's centroid buffered to a
    # radius drawn from the historical parcel-size distribution, never beyond the cap.
    return parent.centroid.buffer(min(parent.geometry.area.median() ** 0.5, max_radius_m))

def spatial_knn(gdf: gpd.GeoDataFrame, column: str, k: int, seed: int) -> gpd.GeoDataFrame:
    rng = np.random.default_rng(seed)
    filled = gdf.copy()
    missing = gdf[column].isna()
    for idx in gdf.index[missing]:
        dist = gdf.geometry.distance(gdf.geometry.loc[idx])
        neighbours = dist[~missing].nsmallest(k).index
        weights = 1.0 / (dist.loc[neighbours] + 1e-9)
        choice = rng.choice(neighbours, p=(weights / weights.sum()).values)
        filled.loc[idx, column] = gdf.loc[choice, column]
    return filled

For undefined or mismatched projections the crs_normalize strategy reprojects to the pipeline’s canonical CRS and refuses to proceed if a datum shift exceeds the contract tolerance — the same coordinate reference system drift guard the contract layer enforces at every interface. Continuous attributes use a distribution-preserving strategy: sample from a truncated Gaussian or Beta fitted to the valid subset, bounded so the injected noise stays inside the differential privacy budget the privacy layer accounts for.

Stage 4 — Validate

No fallback output reaches a consumer without passing the same checks as primary-generation output. Topology is validated with the GEOS/PostGIS predicates documented in the PostGIS Geometry Validation Functions, distributions with a two-sample test, and spatial structure with Moran’s I against the source band declared in the contract, which derives its tolerances from the ISO 19157-1 data-quality metrics.

python
import numpy as np
from scipy.stats import ks_2samp

def validate_fallback(gdf, source_attr: np.ndarray, synth_attr: np.ndarray,
                      fallback_rate: float, *, max_rate=0.05, alpha=0.05) -> None:
    if fallback_rate > max_rate:
        raise ValueError(f"FALLBACK_BUDGET_EXCEEDED: {fallback_rate:.3f} > {max_rate}; quarantine batch.")
    if (~gdf.geometry.is_valid).any():
        raise ValueError("TOPOLOGY_FAILURE: fallback produced invalid geometry.")
    _, p = ks_2samp(source_attr, synth_attr)
    if p < alpha:
        raise ValueError(f"DISTRIBUTION_DRIFT: KS p={p:.4f} < {alpha}; imputed values diverge from source.")

Each executed fallback also emits an immutable audit record — original state, strategy id and version, seed, output checksum, and a confidence score — appended to a hash-chained ledger. Records whose confidence falls below 0.75 are routed to a manual-review queue and suppressed from any ML training set rather than promoted.

Verification: A CI Assertion That Silent Imputation Cannot Pass

A green build is only trustworthy if a batch that breaches the fallback budget or imputes an invalid geometry turns it red. The pytest cases below feed the router known-bad input and assert each gate raises.

python
import numpy as np
import geopandas as gpd
import pytest
from shapely.geometry import Polygon
from router import validate_fallback


def test_budget_gate_quarantines_high_fallback_rate():
    gdf = gpd.GeoDataFrame(geometry=[Polygon([(0, 0), (1, 0), (1, 1), (0, 1)])], crs="EPSG:4326")
    src = np.random.default_rng(0).normal(size=500)
    with pytest.raises(ValueError, match="FALLBACK_BUDGET_EXCEEDED"):
        validate_fallback(gdf, src, src, fallback_rate=0.20)   # 20% > 5% budget


def test_distribution_gate_flags_imputation_drift():
    rng = np.random.default_rng(7)
    source = rng.normal(0, 1, 5000)
    imputed = rng.normal(0, 0.05, 5000)                        # collapsed by naive fill
    gdf = gpd.GeoDataFrame(geometry=[Polygon([(0, 0), (1, 0), (1, 1), (0, 1)])], crs="EPSG:4326")
    with pytest.raises(ValueError, match="DISTRIBUTION_DRIFT"):
        validate_fallback(gdf, source, imputed, fallback_rate=0.01)

Run these as their own job so the gates are exercised on every commit, not only when a batch happens to arrive with gaps.

Edge Cases and Gotchas

Null Island as a fallback sink. The most dangerous fallback is the implicit one: a generator that defaults a missing coordinate to Point(0, 0). The geometry is valid, so a topology gate never catches it, yet a tight knot of features off the coast of Africa silently poisons every area-normalised statistic. Treat 0, 0 as a sentinel for detection, never as a fallback output, and assert no executed strategy can produce it unless the study area legitimately spans the prime meridian and equator.

Antimeridian geometries from buffer fallbacks. A centroid-and-buffer fallback applied to a parent unit near ±180° longitude can emit a polygon that wraps the wrong way around the globe, exploding its bounding box to the full map width while still passing ST_IsValid. Split any ring whose longitudinal span exceeds 180° into a MultiPolygon at the dateline before the validation stage runs, or the distribution gate will see a wildly inflated extent.

Compounding fallbacks on retry. Because pipeline retries replay batches, a non-idempotent strategy applies a fallback on top of a previous fallback, drifting further from source each time. Seeding from immutable batch state and writing the seed into the audit ledger makes execution idempotent; a retry recomputes the identical substitution rather than stacking a new one.

Frequently Asked Questions

When should a fallback fire versus letting the batch fail outright?

A fallback fires only for attributes the contract marks as conditionally required or safely synthesizable, and only while the batch-level fallback rate stays under the declared budget (5% in the example above). Mandatory attributes with no valid neighbour to sample from, or a batch that breaches the budget, must fail the run and quarantine — a fallback is a safety valve, not a substitute for valid source generation.

How do I keep fallback output reproducible across CI runs?

Derive every stochastic strategy’s seed from immutable batch state — the batch ID, the record hash, and the strategy version — and persist that seed in the audit record. Two runs of the same commit then produce byte-identical substitutions, which is what lets a reproducibility test and a privacy attestation pass deterministically.

Won’t imputing missing attributes distort the statistics the realism layer scores?

It can, which is why the validation stage gates on it. A two-sample Kolmogorov–Smirnov test on continuous attributes and a Moran’s I check on spatial structure both run against the contract’s source tolerance band before promotion; if a fallback flattens autocorrelation or shifts the distribution beyond the band, the gate raises DISTRIBUTION_DRIFT and the batch is quarantined rather than scored.

Can a fallback leak real locations or PII?

Only if you let it sample raw high-precision source coordinates. Apply spatial generalization — grid aggregation or Voronoi tessellation — before the execute stage when source data contains precise locations, and bound any noise-injection strategy inside the declared (ε,δ)(\varepsilon, \delta) budget so an imputed attribute can never reconstruct a real-world point.