Stabilizing GAN Training with DP-SGD for Coordinates

When you train a coordinate GAN under DP-SGD, per-sample gradient clipping and calibrated noise routinely flatten the very spatial structure the model is supposed to reproduce, so the challenge is spending a $(\varepsilon, \delta)$ budget that provides a real privacy guarantee while leaving the point pattern’s second-order structure intact.

Part of GAN-Based Spatial Data Generation: that page covers the generator and critic architecture in general; this page drills into training that architecture under a formal privacy guarantee, where the clip norm and noise multiplier are not hyperparameters you tune for accuracy alone but the knobs that trade geographic fidelity against a provable bound on what any adversary can learn about a single input coordinate.

Root Cause: Why DP-SGD Fights Spatial Structure

DP-SGD makes each training step differentially private by bounding and then perturbing the contribution of any single training example. Two operations do this. First, every per-sample gradient is clipped to a maximum L2 norm $C$ :

\bar{g}_i = g_i \cdot \min\!\left(1, \frac{C}{\lVert g_i \rVert_2}\right)

Second, Gaussian noise scaled to that clip norm is added to the summed gradient before the optimizer step:

\tilde{g} = \frac{1}{B}\left(\sum_{i} \bar{g}_i + \mathcal{N}\!\left(0, \sigma^2 C^2 \mathbf{I}\right)\right)

where $B$ is the batch (lot) size and $\sigma$ the noise multiplier. Composed over many steps with subsampling, this yields an $(\varepsilon, \delta)$ bound: informally, the presence or absence of any one coordinate changes the output distribution by at most a factor $e^{\varepsilon}$ , except with probability $\delta$ .

The problem for spatial data is that both operations attack structure the model needs. Clipping caps the influence of the rare, high-gradient samples — which in a heavy-tailed geographic distribution are exactly the peripheral and boundary locations that define the pattern’s extent and its heavy-tailed periphery. Clip too aggressively and the generator learns only the dense cores. The added noise, meanwhile, is isotropic in gradient space but the loss surface for coordinate data is highly anisotropic: a small parameter perturbation that is negligible for one coordinate axis can shift the generated distribution sharply along a spatial gradient, smearing tight clusters into blur. The net effect is a privacy mechanism whose two levers both push toward the same failure — loss of spatial detail — which is why naive DP-SGD on coordinates so often produces a privacy-compliant but geographically useless generator, and frequently one that also mode-collapses (see diagnosing mode collapse in spatial GANs).

The stabilization strategy is to normalize coordinates so per-sample gradient norms are comparable across the extent, set the clip norm from the observed gradient-norm distribution rather than a guessed constant, and pick the smallest $\sigma$ that meets the target $\varepsilon$ under a tight accountant.

Every step clips per-sample gradients into the norm ball C, adds noise scaled by σC, and applies the result; the accountant converts (σ, q, t) into a cumulative ε that must stop the run before it crosses the budget ceiling.

Prerequisite Check: Confirm the Privacy Path Is Wired

Before tuning, confirm the optimizer actually computes per-sample gradients (not batch-averaged ones, which give no privacy) and that an accountant is attached. Pin the toolchain.

python
# requirements.txt — pin majors; let patch releases float
numpy==1.26.*
torch==2.*
opacus==1.*        # per-sample gradients, clipping, RDP accountant

python
import torch
from opacus import PrivacyEngine

# A private optimizer must expose per-sample grad support and an accountant.
def assert_dp_wired(privacy_engine: PrivacyEngine, optimizer) -> None:
    assert hasattr(optimizer, "max_grad_norm"), "clip norm not set: no per-sample clipping"
    assert privacy_engine.accountant is not None, "no accountant: epsilon is not being tracked"
    # A zero or absent noise multiplier means epsilon is unbounded (infinite).
    assert getattr(optimizer, "noise_multiplier", 0.0) > 0.0, "sigma=0 -> no privacy guarantee"

# If get_epsilon() raises or returns inf, the run has no valid (epsilon, delta) bound.

If noise_multiplier is zero or the accountant is missing, the run trains normally but carries no privacy guarantee — the most dangerous silent failure, because the artifact looks private and is not.

Fix: Normalize, Set the Clip Norm from Data, Then Spend the Budget

The stabilization sequence is: normalize coordinates so gradient norms are comparable, choose the clip norm from the observed per-sample gradient-norm distribution, and select the smallest noise multiplier that reaches the target $\varepsilon$ .

Step 1 — Equal-area normalization of coordinates

Project to an equal-area CRS and standardize, so no axis dominates the gradient norm and clipping does not preferentially discard one direction of spatial spread. Latitude/longitude in EPSG:4326 distorts distances with latitude; EPSG:6933 keeps area comparable.

python
import numpy as np
from pyproj import Transformer

def normalize_coords(lon: np.ndarray, lat: np.ndarray) -> tuple[np.ndarray, dict]:
    """Reproject to equal-area metres, then zero-mean/unit-std per axis."""
    tf = Transformer.from_crs("EPSG:4326", "EPSG:6933", always_xy=True)
    x, y = tf.transform(lon, lat)
    xy = np.column_stack([x, y]).astype("float64")
    mu, sd = xy.mean(0), xy.std(0)
    stats = {"mu": mu, "sd": sd}            # persist to invert after generation
    return (xy - mu) / sd, stats

Step 2 — Set the clip norm from the gradient-norm distribution

Guessing $C$ wastes budget: too large and you inject more noise than needed, too small and you discard the peripheral gradients. Measure the per-sample gradient-norm distribution on a non-private warmup batch and set $C$ near its median.

python
import torch

def calibrate_clip_norm(model, loss_fn, batch, quantile: float = 0.5) -> float:
    """Median per-sample gradient L2 norm — a data-driven clip that keeps the tail."""
    norms = []
    for sample in batch:                       # one example at a time -> true per-sample grad
        model.zero_grad(set_to_none=True)
        loss_fn(model, sample.unsqueeze(0)).backward()
        g = torch.cat([p.grad.flatten() for p in model.parameters() if p.grad is not None])
        norms.append(g.norm().item())
    return float(torch.tensor(norms).quantile(quantile))

Step 3 — Attach the engine and pick σ for the target ε

Solve for the smallest noise multiplier that reaches the target $\varepsilon$ at a fixed $\delta$ over the planned number of steps, using the accountant rather than a hand rule. Bind $\delta$ to less than the inverse dataset size so the guarantee is meaningful; this is the same budgeting discipline the privacy-preserving generation frameworks apply across releases.

python
import torch
from opacus import PrivacyEngine

def make_private(model, optimizer, data_loader, *,
                 target_epsilon: float, target_delta: float, epochs: int, max_grad_norm: float):
    engine = PrivacyEngine(accountant="rdp")   # Renyi DP -> tight composition
    model, optimizer, data_loader = engine.make_private_with_epsilon(
        module=model,
        optimizer=optimizer,
        data_loader=data_loader,
        target_epsilon=target_epsilon,   # e.g. 8.0
        target_delta=target_delta,       # e.g. 1e-5 < 1/N
        epochs=epochs,
        max_grad_norm=max_grad_norm,     # C from Step 2
    )
    return engine, model, optimizer, data_loader

# After training, report the SPENT budget, not the target:
#   eps = engine.get_epsilon(delta=target_delta)  # must be <= target_epsilon

Apply DP-SGD to the critic only in the common setup: by the post-processing property of differential privacy, if the critic is trained privately then the generator — which only ever sees the critic’s private gradients, never the raw data — inherits the guarantee without needing its own noise. This halves the fidelity cost. For how DP-SGD compares to output-perturbation and PATE-style mechanisms on spatial data, see differential privacy mechanisms compared.

Verification Step: Assert the Budget and the Structure Together

A DP run must be gated on two things at once: the spent $\varepsilon$ is within budget, and the spatial structure survived. Gating on privacy alone ships a useless generator; gating on fidelity alone ships a non-private one.

python
import numpy as np

def test_dp_run_is_private_and_faithful(engine, generator, ref_points, *,
                                         target_epsilon=8.0, target_delta=1e-5, seed=42):
    # 1. Privacy: spent epsilon must not exceed the budget at the bound delta.
    eps = engine.get_epsilon(delta=target_delta)
    assert np.isfinite(eps) and eps <= target_epsilon + 1e-6, f"budget exceeded: eps={eps:.2f}"

    # 2. Structure: nearest-neighbour distance distribution must track the reference.
    z = np.random.default_rng(seed).standard_normal((4096, generator.latent_dim))
    fake = generator.sample(z)
    def nnd(p):
        from scipy.spatial import cKDTree
        d, _ = cKDTree(p).query(p, k=2)
        return np.sort(d[:, 1])
    a, b = nnd(fake), nnd(ref_points)
    # Kolmogorov-Smirnov style gap on the NND CDFs, resampled to equal length.
    grid = np.linspace(0, max(a.max(), b.max()), 256)
    gap = np.abs(np.searchsorted(a, grid) / len(a) - np.searchsorted(b, grid) / len(b)).max()
    assert gap < 0.15, f"DP noise destroyed clustering structure: NND gap {gap:.3f}"

The nearest-neighbour-distance gap is the tell-tale for over-noising: as $\sigma$ rises past the usable range, tight clusters diffuse and the generated NND distribution shifts right, tripping this gate well before a human notices the map looks wrong.

Edge Cases & Gotchas

Clipping silently deletes the antimeridian tail. Coordinates straddling ±180° longitude produce two far-apart clusters in raw EPSG:4326, so per-sample gradients for the seam-crossing points are large and are exactly what clipping truncates first — the model then learns only one side. Reproject into the equal-area metric CRS (Step 1) before any gradient is computed, so the seam becomes an ordinary interior region and its gradients are unexceptional.

Null Island as a normalization sink. Records defaulted to (0, 0) from a failed geocode form a spike at the origin that dominates the mean and inflates per-axis std, so normalization shrinks the real extent toward zero and clipping then treats genuine peripheral points as outliers. Filter (0, 0) (and other sentinel coordinates) before computing normalization statistics, and assert the filtered fraction is below a threshold.

δ set larger than 1/N voids the guarantee. Because $\delta$ is the probability the $\varepsilon$ bound simply fails, choosing $\delta \ge 1/N$ for a dataset of $N$ records permits a mechanism that reproduces whole records with non-negligible probability while still “passing.” Always set $\delta < 1/N$ (commonly $10^{-5}$ or smaller), and record both $\varepsilon$ and $\delta$ with the artifact — reporting $\varepsilon$ without its $\delta$ is meaningless.

Frequently Asked Questions

Should I apply DP-SGD to the generator, the critic, or both?

Apply it to the critic only in the standard setup. The generator never touches the raw training data — it learns only through the critic's gradients — so by the post-processing property of differential privacy, a privately trained critic confers the same guarantee on the generator for free. Adding noise to the generator as well spends budget twice and degrades fidelity for no additional protection.

How do I pick the clip norm C without wasting budget?

Measure the per-sample gradient-norm distribution on a short non-private warmup and set C near its median. A clip much larger than typical norms forces you to add proportionally more noise for the same privacy, while a clip much smaller truncates the heavy-tailed peripheral gradients and drops the pattern's extent. The median is a robust starting point; tune within the interquartile range against the fidelity gate.

Why report both epsilon and delta instead of just epsilon?

Delta is the probability that the epsilon bound fails to hold at all, so an epsilon value is only interpretable alongside its delta. A small epsilon paired with a large delta offers weak protection because the guarantee can break with non-negligible probability. Set delta below the inverse of the dataset size and always record the pair with the artifact.

My DP generator mode-collapsed — is that the noise or a separate bug?

DP noise on the critic can trigger or worsen mode collapse, because it can drown out the batch-diversity signal the critic uses to punish repetition. Diagnose it with spatial-entropy and coverage metrics rather than the loss, and tune the noise multiplier alongside any anti-collapse remedy such as minibatch discrimination rather than treating the two problems in isolation.

GAN-Based Spatial Data Generation — the parent workflow this page trains under a formal privacy guarantee.
Diagnosing Mode Collapse in Spatial GANs — how privacy noise interacts with, and can trigger, collapse.
Privacy-Preserving Generation Frameworks — the budgeting discipline that governs ε across releases, not just one run.
Differential Privacy Mechanisms Compared for Spatial Data — where DP-SGD sits against output perturbation and PATE for coordinates.
Adversarial Privacy Testing for Synthetic Spatial Data — empirically stress the guarantee the accountant claims.