Normalising Density Surfaces Across Releases

Two consecutive density releases are compared cell by cell and every cell has changed. The underlying events barely moved, the bandwidth is the same, the grid is anchored — and the numbers are all different, because the normalisation rescaled the whole surface.

Part of Density Mapping & Heat Generation: that page covers making a single surface reproducible. This one is about making two of them comparable, which is a stronger requirement and a different mistake.

Root Cause: Adaptive Scaling Is a Function of the Data

Nearly every density pipeline ends with a normalisation step, and nearly all of them are adaptive: scale to the maximum, scale to the ninety-ninth percentile, standardise to zero mean and unit variance. All three are excellent for producing a single readable map, and all three make consecutive releases incomparable.

The mechanism is simple. An adaptive scale is computed from the release, so the same absolute density maps to a different scaled value whenever the release’s own distribution moves. Add one unusually dense cell and the p99 shifts, and every other cell’s value changes to accommodate it — including cells whose underlying events did not change at all.

That produces three symptoms that look like separate problems:

  • Everything changed. A cell-by-cell diff between releases is dominated by the rescaling, so no per-cell comparison is meaningful.
  • A change in one place moves values everywhere. New development in one district shifts the p99, which changes every cell in the country.
  • Trends are unmeasurable. A genuine increase in density across the whole extent produces no change at all in a max-scaled surface, because the maximum increased too.
Reported change in unchanged cells under five normalisation schemes Five normalisation schemes run along the horizontal axis. For each there are two bars, both expressed as a percentage of the scaled surface's range. The first bar is the average change reported in cells whose underlying events did not move at all — spurious change, and it should be zero. The second is the average change reported in the cells that genuinely more than doubled — real signal, and it should be large. Publishing the absolute surface and scaling against an anchor taken from a fixed reference both report exactly zero spurious change while preserving the real signal in full. Scaling to the release's own ninety-ninth percentile, to its own maximum, or standardising it all report visible change in cells where nothing happened, because the divisor moved when the changed cells moved. The note underneath states the consequence: a cell-by-cell diff between two adaptively scaled releases mixes real change with rescaling artefact, and there is no way to separate them after the fact. Three of the five report change in cells where nothing happened 0% 10% 20% 30% reported change (% of range) 0 absolute 0 anchored p99 4.4 scale to p99 4.1 scale to max 2.1 z-score real change, in cells that moved spurious change, in cells that did not 400 cells, 24 of them genuinely increased, everything else identical between the two releases, fixed seed. A cell-by-cell diff between two adaptively scaled releases mixes real change with rescaling artefact, and nothing downstream can separate them afterwards — the information needed to do so was destroyed by the divisor.
The same underlying change under four normalisation schemes: three of them report a change in cells where nothing happened, and one of them reports no change where something did.

Prerequisite Check: Establish Which Question the Surface Answers

Before choosing a scheme, decide what a consumer will do with the numbers, because the schemes are not interchangeable.

python
INTENT = {
    "cartographic":   "one map, read by eye, no cross-release comparison",
    "comparative":    "two releases compared cell by cell",
    "absolute":       "values used as densities in a downstream calculation",
}

Only the first is served by adaptive scaling, and it is the intent most pipelines were originally built for — which is why the normalisation is usually inherited rather than chosen. The second and third both require the scale to be independent of the release, and they differ in whether the units have to mean anything.

Fix: Anchor the Scale Outside the Release

1 — Publish the unnormalised surface, always

python
def emit(surface, contract: dict) -> dict:
    """The absolute surface is the artifact; scaled variants are derived views."""
    return {
        "density": surface,                          # events per km², absolute
        "units": "events_per_km2",
        "bandwidth_m": contract["bandwidth_m"],
        "cell_m": contract["cell_m"],
        "crs": contract["crs"],
    }

This alone resolves most of the problem, and it is often resisted because the absolute surface is harder to render. That is a rendering concern, and the answer is to derive the display scaling at render time rather than to bake it into the artifact — which is the same argument as shipping unsnapped coordinates with a separately versioned match.

2 — Where a scaled variant is needed, anchor it to a reference

python
from dataclasses import dataclass


@dataclass(frozen=True)
class ScaleAnchor:
    reference_release: str      # the artifact hash the anchor was computed from
    p99: float                  # events per km² at the reference's 99th percentile
    computed_at: str


def scale(surface, anchor: ScaleAnchor):
    """Scale by a constant taken from a named reference, not from this release."""
    return surface / anchor.p99

The anchor is a stored constant with a provenance record, not a computation. Two releases scaled against the same anchor are directly comparable, and a cell whose value is 1.4 means the same density in both.

3 — Re-anchor deliberately, and version the anchor

An anchor eventually drifts out of usefulness — the population genuinely changed, and a scale set three years ago compresses everything into the bottom of the range. Re-anchoring is legitimate; doing it silently is not.

python
def should_reanchor(surface, anchor: ScaleAnchor, drift: float = 0.35) -> bool:
    """Flag when the current release's p99 has moved far from the anchor's."""
    current = percentile(surface, 99)
    return abs(current - anchor.p99) / anchor.p99 > drift

When it fires, the correct action is a new anchor version and a release note saying so — not an adjustment. A consumer comparing across the re-anchor point needs to know that the scale changed, and the only thing that tells them is the anchor version travelling with the surface.

The life of a density scale anchor Four stages read left to right. In the first the anchor is established: a percentile is computed once from a named reference release and stored as a constant with the reference's hash and the date it was taken. In the second it is held: many releases are scaled by that same constant, so their cell values are directly comparable and a cell whose underlying events did not change carries an identical number. In the third it drifts: the population genuinely moves and the current release's own percentile pulls away from the stored anchor, which a threshold check detects and reports without changing anything. In the fourth it is superseded: a new anchor version is issued with its own reference and date, the old value is retained rather than overwritten, and the release note records that the scale changed at that point. The note underneath states the rule the diagram enforces: the anchor may change, but only by version, because a consumer comparing across the boundary needs to know a comparison stopped being valid there. The anchor is a versioned constant, not a computation establish percentile taken once from a named release stored with its hash hold many releases scaled by the same constant values comparable drift population moves release p99 pulls away threshold check reports supersede new anchor version old value retained release note says so what stays fixed the numeric scale value, for the anchor's whole life the reference release hash it was computed from the date it was taken what a re-anchor must produce a new version identifier travelling with the surface the previous anchor kept, not overwritten a release note naming the release the scale changed at The anchor may change — a scale set years ago eventually compresses a grown population into the bottom of its range. It may only change by version. A consumer comparing across the boundary needs to know that a comparison stopped being valid there, and the version identifier is the only thing that tells them.
An anchor's life: fixed across many releases, drifting slowly as the population moves, and superseded explicitly rather than adjusted.

Verification Step: Assert Comparability, Not Just Correctness

python
def test_unchanged_cells_are_unchanged(previous, current, unchanged_mask, tol=1e-9):
    """Cells whose underlying events did not change must have identical values."""
    delta = abs(current[unchanged_mask] - previous[unchanged_mask]).max()
    assert delta < tol, f"a cell with no event change moved by {delta:.6f}"


def test_scale_anchor_is_recorded(manifest):
    assert manifest["density"]["scale_anchor"]["reference_release"], (
        "a scaled surface with no anchor reference is not comparable to anything"
    )


def test_absolute_surface_is_published(release):
    assert "density" in release and release["units"] == "events_per_km2"


def test_reanchor_is_flagged(surface, anchor, manifest):
    if should_reanchor(surface, anchor):
        assert manifest["density"]["scale_anchor"]["version"] != anchor.version, (
            "the p99 has drifted past the re-anchor threshold and the anchor did not change"
        )
Percentile drift against a fixed scale anchor across a release series The horizontal axis is the release number across twenty-six consecutive releases. The vertical axis is density at the ninety-ninth percentile, in events per square kilometre. The flat line is the stored anchor, taken once from the first release and never recomputed. The rising line is each release's own percentile, drifting upward at roughly two percent per release as the modelled population grows, with release-to-release noise on top. A shaded band around the anchor marks the thirty-five percent drift tolerance. For most of the series the current percentile sits inside the band, which is exactly the situation the anchor exists for: the population is changing, and the scale deliberately is not, so releases stay comparable. Partway through the series the percentile leaves the band, and that release is marked as the point where the drift check fires. The note underneath is explicit that the check reports rather than acts: firing means a re-anchor decision is due, with a new version and a release note, not that the scale should quietly adjust itself. The check fires a decision, not an adjustment 0 5 10 15 20 25 50 100 150 200 release number p99 density (events / km²) stored anchor drift check fires at release 16 ±35% tolerance band 26 releases with roughly 2% population growth each and release-to-release noise, fixed seed. Inside the band is the situation the anchor exists for: the population changes and the scale deliberately does not, so releases stay comparable. Outside it, the correct response is a new anchor version and a release note — not a quiet adjustment, which would silently break every comparison spanning the change.
A slowly growing population against a fixed anchor: the drift check reports when a re-anchor decision is due rather than adjusting the scale itself.

The first assertion is the one that would have caught the original symptom, and it is only expressible because the generator knows which cells changed. That is the same argument as the ground-truth stop layer and the on-network flag: the producer holds information the consumer cannot recover, and a check built on it catches things no consumer-side check can.

Edge Cases & Gotchas

The reference release is withdrawn. The anchor outlives it — it is a stored constant, not a live dependency — but the provenance record now points at something unavailable. Keep the anchor’s numeric value and its reference hash regardless; a withdrawn reference does not invalidate a scale computed from it, and pretending otherwise forces an unnecessary re-anchor.

Different regions need different scales. A single global anchor compresses a rural surface to nothing. Anchor per region, version each independently, and make the region key part of the anchor identity — but resist per-release regional anchors, which reintroduces the original problem at a finer granularity.

Log scaling. Widely used for density and entirely compatible with anchoring, provided the log is taken before the anchored division rather than after. Taking it after produces a scale whose zero point moves with the anchor, which is the same defect in a subtler form.

A consumer who wants the p99 anyway. Publish it as a statistic of the release alongside the surface. The problem was never computing the percentile — it was using it as the scale.

Why This Keeps Being Rediscovered

Adaptive normalisation is not a careless choice. It is the right choice for the thing most density pipelines were originally built to do, and it becomes wrong later without anybody changing a line of code.

The first release of a density surface is almost always cartographic. Somebody wants a map, the map needs to use its colour ramp fully, and scaling to the release’s own percentile is exactly how you get that. It is correct, it is conventional, and it has no downside in a world with one release.

The second release arrives and the requirement silently changes. Now somebody wants to know what moved. The pipeline is unchanged, the surfaces look right individually, and the comparison is meaningless — but nothing failed, no error was raised, and the numbers are all plausible. That is the whole difficulty: an incomparable pair of surfaces looks exactly like a comparable pair, and the only way to tell them apart is to know how the scale was computed.

Three habits make this survivable:

  • Treat “will anyone compare two of these?” as a design question, not a future concern. The answer for anything published on a schedule is yes, and the cost of anchoring from the start is a stored constant.
  • Make the scale visible in the artifact. A surface that carries its units and its anchor version can be checked; one that carries neither cannot be, by anybody, ever.
  • Separate the map from the data. The rendering can scale however it likes, per release, per region, per viewer. The problem only appears when the display choice is baked into the stored values.

None of that is specific to density. It is the same argument as versioning a schema or shipping unsnapped coordinates: a transformation applied before storage destroys information the consumer needed, and the fix is always to move the transformation later rather than to make it smarter.