Conditioning a Spatial GAN on a Covariate Raster

A land-use raster is concatenated onto the generator input, training converges, the samples look right — and the output density is the same over water as over the town centre. The condition was supplied and ignored.

Part of GAN-Based Spatial Generation: this is the specific failure where a conditional model trains successfully as an unconditional one, which is both common and quiet.

Root Cause: Nothing in the Objective Requires the Condition to Be Used

A generator concatenated with a covariate channel is free to ignore that channel. Zero the weights on it and the model is a perfectly good unconditional generator, producing samples the discriminator accepts, with a loss indistinguishable from a conditional model that works.

Whether it uses the condition depends entirely on whether the discriminator can tell. If the discriminator sees only the sample, then a plausible sample in the wrong place is indistinguishable from a plausible sample in the right place, and the gradient carries no information about location. The generator learns the marginal distribution of patterns and nothing about where they belong.

This is why the fix is almost never in the generator. Three arrangements, in increasing order of how strongly they force the condition to matter:

  • Concatenate to the generator only. The condition is available and unenforced. This is the arrangement that produces the symptom, and it is the default anyone writes first.
  • Concatenate to both generator and discriminator. The discriminator now sees the pair, so a mismatched pair can be rejected — but early in training it is easier for the discriminator to discriminate on sample realism alone, and the conditioning signal is weak and often stays weak.
  • Projection discriminator. The condition enters through an inner product with the discriminator’s feature vector, so the compatibility of sample and condition is a distinct, first-class term in the score rather than something the network may or may not learn to extract.
Three places a covariate condition can enter a spatial GAN Three arrangements shown side by side, each with a generator on the left and a discriminator on the right. In the first the covariate is concatenated onto the generator input only. The discriminator never sees it, so a sample placed in entirely the wrong location scores exactly as well as one placed correctly, and nothing in the objective requires the generator to use the channel at all. In the second the covariate is concatenated onto both. The discriminator can now reject a mismatched pair, but it is not obliged to learn to, and early in training discriminating on sample realism alone is easier, so the conditioning signal is weak and often stays weak. In the third the covariate is embedded and combined with the discriminator's feature vector through an inner product, so sample-condition compatibility is a separate additive term in the score rather than something the network may or may not extract. Under each arrangement a strength bar shows how firmly the condition is enforced: none, weak, and strong respectively. The note underneath states the diagnosis rule: if the discriminator cannot see the condition, the generator is unconditional no matter what its input looks like. The condition only matters where the loss can see it condition → G only G sees the covariate D sees the sample alone wrong place scores the same condition enforced by the loss none condition → G and D both see the covariate D may learn to use it realism is the easier signal condition enforced by the loss weak projection discriminator condition embedded inner product with features compatibility is its own term condition enforced by the loss strong The diagnosis rule is short: if the discriminator cannot see the condition, the model is unconditional no matter what the generator's input looks like. A converging loss curve tells you nothing here — an unconditional generator trained with a covariate channel it ignores converges perfectly well, and its samples are individually plausible. Only a response sweep distinguishes the two.
Where the condition can enter, and how strongly each arrangement forces the generator to use it.

Prerequisite Check: Is the Covariate Actually Predictive?

Before diagnosing a model, check that the condition carries signal at the resolution being used. A covariate that does not predict the target in the real data cannot be learned from it, and weeks are lost to models that were asked to learn something absent.

python
def conditional_signal(target_counts, covariate, bins=8) -> dict:
    """Mutual information between covariate band and target count, in nats."""
    edges = quantile_edges(covariate, bins)
    band = digitize(covariate, edges)
    joint, px, py = histogram2d_normalised(band, discretise(target_counts))
    mi = sum(
        joint[i][j] * math.log(joint[i][j] / (px[i] * py[j]))
        for i in range(len(px)) for j in range(len(py))
        if joint[i][j] > 0
    )
    return {"mi_nats": mi, "usable": mi > 0.02}

Two failure modes it catches. A covariate with essentially zero mutual information is not a conditioning failure waiting to happen — it is a covariate to drop. And a covariate with high mutual information at 1 km but none at 50 m has been supplied at the wrong resolution: the model is being asked to use detail that carries no signal, and the resolution mismatch will show up as noise rather than as an error.

Fix: Put the Condition Where the Loss Can See It

1 — Project the condition into the discriminator’s score

python
class ProjectionDiscriminator(nn.Module):
    """Score = unconditional realism + inner product of features with embedded condition."""

    def __init__(self, feat_dim: int, cond_channels: int):
        super().__init__()
        self.trunk = ConvTrunk(out_dim=feat_dim)
        self.psi = nn.Linear(feat_dim, 1)               # realism term
        self.embed = CondEncoder(cond_channels, feat_dim)  # condition → same space

    def forward(self, sample, condition):
        h = self.trunk(sample)                           # (B, feat_dim)
        c = self.embed(condition)                        # (B, feat_dim)
        return self.psi(h).squeeze(-1) + (h * c).sum(dim=-1)

The inner product is the whole point. It is a term whose value depends on sample and condition jointly, so a real sample paired with the wrong covariate patch scores low even though the sample itself is perfectly realistic — and that is exactly the gradient the generator needs.

2 — Train against mismatched pairs explicitly

python
def discriminator_loss(D, real, cond, fake):
    d_real = D(real, cond)                    # real sample, right condition  → high
    d_fake = D(fake, cond)                    # fake sample, right condition  → low
    d_swap = D(real, roll(cond, shifts=1))    # real sample, wrong condition  → low
    return (relu(1 - d_real) + relu(1 + d_fake) + relu(1 + d_swap)).mean()

The third term is cheap — one extra forward pass on a shuffled condition batch — and it is what turns “the discriminator could learn to use the condition” into “the discriminator cannot achieve a low loss without using it”. In practice adding this one line resolves the majority of silently-unconditional models.

3 — Normalise the covariate so the scale does not swamp the noise

python
def prepare_covariate(raster, stats):
    """Standardise per band against fixed training statistics, not per batch."""
    return (raster - stats.mean) / stats.std

Fixed statistics, stored alongside the model. Per-batch standardisation makes the condition mean something different in every batch, which is a subtler version of the same problem as adaptive density scaling — the number no longer means one thing.

Output response to a swept condition under three arrangements The horizontal axis sweeps the covariate from its lowest level to its highest with the generator's noise vector held constant, so every change in the output comes from the condition. The vertical axis is generated output intensity, normalised so that one is the average. A dashed line shows the relation measured in the observed data, rising steeply and slightly faster than linearly. Conditioning the generator alone produces a flat line: the output is the same at every covariate level, which is the signature of a model that ignored the channel entirely. Concatenating to both generator and discriminator produces a line that rises, but far too shallowly, reaching only part of the observed range. The projection discriminator tracks the observed relation closely across the whole sweep. Each series is annotated with its response spread, the relative range of the output across the sweep, which is the single number the verification test asserts on. The note underneath gives the threshold and what falling below it means. Sweep the condition with the noise fixed — the flat line is the failure 0.00 0.25 0.50 0.75 1.00 0.0 0.5 1.0 1.5 2.0 covariate level (swept, noise held fixed) output intensity (1 = mean) observed relation spread 0.10 spread 0.51 spread 1.58 condition → G only condition → G and D projection discriminator The test asserts a response spread above 0.15. Generator-only conditioning scores near zero here — the output does not move at all — and that is the silently-unconditional model, which no loss curve distinguishes from a working one. A spread that is large but uncorrelated with the observed relation is a different failure and needs the sign check, not more training.
Measured response to the condition under three arrangements: how far output intensity moves when the covariate band is swept, against how far it should move.

Verification Step: Sweep the Condition and Measure the Response

The decisive test is not a loss curve. Hold the noise vector fixed, sweep the covariate, and measure whether the output moves.

python
def response_curve(G, z_fixed, covariate_levels) -> list[float]:
    """Output intensity as a function of the condition, with noise held constant."""
    return [float(G(z_fixed, level).sum()) for level in covariate_levels]


def test_generator_responds_to_the_condition(G, z_fixed, levels, reference):
    got = response_curve(G, z_fixed, levels)
    spread = (max(got) - min(got)) / (sum(got) / len(got))
    assert spread > 0.15, f"output barely moves with the condition (spread {spread:.3f})"


def test_response_has_the_right_sign(G, z_fixed, levels, reference):
    got = response_curve(G, z_fixed, levels)
    assert spearman(got, reference) > 0.7, "response is uncorrelated with the observed relation"


def test_no_output_where_the_covariate_forbids_it(G, z, water_mask, tol=1e-3):
    sample = G(z, water_mask_condition)
    assert sample[water_mask].mean() < tol, "features generated over water"

The first assertion catches the silent-unconditional case directly, and it is the one that would have caught the original symptom on the first training run. The third is the domain check: some covariate values are hard constraints, not tendencies, and a model that puts a building in a lake has failed in a way no aggregate statistic reports.

Reading a conditional generator's response sweep A matrix with four sweep shapes as rows. A flat response — the output identical at every covariate level — means the discriminator never sees the condition, and the fix is to move the condition into the discriminator through projection rather than to train longer. A weak response that rises in the right direction but covers only part of the observed range means the conditioning signal is present but under-weighted, and the fix is to add mismatched pairs to the discriminator loss so a low loss becomes unreachable without using the condition. An inverted response, moving in the wrong direction, almost always means the covariate was supplied with a different orientation or normalisation at inference than at training, and the fix is in the data path rather than the model. A collapsed response, where the output moves with the condition but not with the noise, means the generator has become a deterministic function of the covariate, and the fix is to reduce the conditioning weight. The note underneath points out that only the first two are conditioning problems at all. Four sweep shapes, four different fixes sweep shape what it means where to change it flat — no movement D never sees the condition unconditional model projection discriminator not more epochs weak — right sign, small range signal present, under-weighted realism dominates mismatched-pair loss term one extra forward pass inverted — wrong direction covariate orientation differs train vs inference the data path not the model collapsed — noise stops mattering generator is deterministic given the covariate lower the conditioning weight run the noise sweep Only the first two rows are conditioning problems. An inverted response is a pipeline defect wearing a modelling costume, and a collapsed one is the opposite failure to the one being diagnosed — which is why the noise sweep is worth running alongside the covariate sweep rather than after somebody notices.
Reading the response sweep: four shapes, what each one means, and which part of the arrangement to change.

Edge Cases & Gotchas

Conditioning collapse. The opposite failure: the generator uses the condition so heavily that the noise vector stops mattering and every sample from a given covariate patch is identical. The same sweep detects it, run the other way — hold the condition fixed and vary the noise. If the output does not move, the model has become a deterministic function of the covariate.

Covariate leakage into privacy. A high-resolution covariate that is itself derived from the training data reintroduces exactly the information a synthetic release was meant to remove, and it does so through a channel that no membership inference test on the samples alone will detect. The condition needs to be public data or an audited derivative.

Class imbalance in the covariate. If ninety percent of the extent is one land-use class, the model sees the other classes rarely and conditions poorly on them. Stratified patch sampling during training fixes this and costs nothing at inference.

Resolution mismatch between condition and output. Upsampling a 1 km covariate to a 50 m output grid supplies detail that is not there. It is not harmful, but it invites the mistake of evaluating the model at 50 m and concluding the conditioning is poor when the input never carried that resolution.

What Conditioning Buys, and What It Does Not

A working conditional generator changes what the release can be used for, and it is worth being precise about the boundary, because the claim is routinely overstated in both directions.

It buys spatial plausibility that no post-hoc filter can. Generating unconditionally and then discarding samples that land in implausible places produces a correct-looking result at a catastrophic acceptance rate wherever the covariate is restrictive, and it silently changes the distribution of everything that survives — the same defect as rejection sampling against a tight constraint, arriving through a different door.

It buys transferability. A model conditioned on land use can be run over a covariate raster for a region it never saw in training, and produce something defensible there. An unconditional model can only reproduce the region it learned. For most release programmes this is the actual reason conditioning is worth the trouble.

It does not buy correctness of the relationship. The generator learns the association present in the training data, including whatever confounding that data carries. If the observed relationship between the covariate and the target is an artifact of how the observations were collected, the model reproduces the artifact faithfully and the response sweep will look excellent.

It does not remove the need for the marginal checks. A well-conditioned model can still have the wrong overall intensity, the wrong cluster structure, or the wrong attribute correlations. Conditioning constrains where things go; it says nothing about how many or what they look like, and those remain separate checks with separate targets.