Building a Realism Scorecard for a Release

The pipeline computes fourteen metrics. Twelve pass, one is borderline, one fails on a dimension nobody downstream uses. Somebody has to decide whether the release ships, and the fourteen numbers do not answer that on their own.

Part of Realism Metrics & Evaluation: that page covers which metrics detect which defects. This one is about assembling them into an artifact a reviewer can act on, and the two ways that assembly is usually done wrong.

Root Cause: A Metric Set Is Not a Verdict

Two failure modes dominate, and they are opposites.

The first is the scatter. Fourteen numbers in a log, each with its own tolerance, no grouping and no summary. A reviewer reads the first three, sees green, and approves; the failure sits at position eleven. Nothing is wrong with any individual number and the artifact is unusable, because it asks the reviewer to do the aggregation and gives them no basis for it.

The second is the average. Somebody collapses the fourteen into a single realism score, usually a weighted mean of normalised deviations. It is immediately readable and it is worse than the scatter, because it lets a catastrophic failure on one dimension be offset by excellent scores elsewhere. A release whose geometry is invalid and whose statistics are perfect scores well, and the one number that mattered has been averaged away.

The resolution is neither: group the metrics by dimension, take the worst verdict within each dimension rather than the mean, and require every dimension to pass independently.

One metric set under three aggregation rules, with the verdict each produces The same fourteen metric results are shown three ways. On the left, an ungrouped scatter: fourteen values in a list, twelve passing, one borderline and one failing. It gives no summary at all, and a reviewer who reads the first few sees green. In the middle, a weighted average of normalised deviations collapses them to a single score of 0.86, which reads as comfortably acceptable — and it is comfortably acceptable only because eleven strong distributional results have paid for a structural failure that no downstream consumer can tolerate. On the right, the metrics are grouped into five dimensions and each dimension takes the worst verdict among its members: four dimensions pass and the structural one fails, so the release fails and the driving metric is named. The three answers are green, green and red for identical inputs. The note underneath states the rule this figure exists to argue for: never average across dimensions, because averaging is exactly the operation that lets a strong dimension pay for a failing one. Identical inputs, three answers: green, green, red ungrouped scatter no summary at all a reviewer reads the first few and sees green no verdict weighted average one score: 0.86 0.86 eleven strong results paid for one structural failure PASS — wrongly worst per dimension five verdicts, one each structural distributional spatial relational integrity structural fails; nothing can pay for it FAIL — correctly Never average across dimensions. Averaging is precisely the operation that lets a strong dimension pay for a failing one, and the dimension most likely to fail is the one with the fewest metrics in it.
The same fourteen metrics under three aggregation rules: only the third refuses to let a strong dimension pay for a failing one.

Prerequisite Check: Assign Every Metric to Exactly One Dimension

python
from dataclasses import dataclass
from enum import Enum


class Dimension(Enum):
    STRUCTURAL = "structural"        # is the geometry well-formed at all
    DISTRIBUTIONAL = "distributional"  # do the marginals match
    SPATIAL = "spatial"              # does the spatial structure match
    RELATIONAL = "relational"        # do the attributes relate correctly
    INTEGRITY = "integrity"          # duplicates, nulls, referential rules


@dataclass(frozen=True)
class Metric:
    name: str
    dimension: Dimension
    value: float
    tolerance: float
    higher_is_better: bool = False

    @property
    def verdict(self) -> str:
        ok = (self.value >= self.tolerance if self.higher_is_better
              else self.value <= self.tolerance)
        margin = abs(self.value - self.tolerance) / max(abs(self.tolerance), 1e-9)
        if not ok:
            return "fail"
        return "borderline" if margin < 0.15 else "pass"

The one-dimension rule is what makes the aggregation meaningful. A metric assigned to two dimensions is counted twice and can rescue a dimension it does not really speak to; a metric assigned to none is a metric nobody looks at.

The borderline verdict earns its place: a metric passing by two per cent of its tolerance carries very different information from one passing by two hundred, and collapsing both to “pass” throws away the only early warning the scorecard produces.

Fix: Aggregate Within Dimensions, Never Across Them

python
def dimension_verdict(metrics: list[Metric], dimension: Dimension) -> dict:
    members = [m for m in metrics if m.dimension is dimension]
    if not members:
        return {"verdict": "not covered", "metrics": []}
    verdicts = [m.verdict for m in members]
    worst = "fail" if "fail" in verdicts else (
        "borderline" if "borderline" in verdicts else "pass")
    return {
        "verdict": worst,
        "driver": next(m.name for m in members if m.verdict == worst),
        "metrics": [{"name": m.name, "value": m.value, "tolerance": m.tolerance,
                     "verdict": m.verdict} for m in members],
    }


def scorecard(metrics: list[Metric]) -> dict:
    dims = {d.value: dimension_verdict(metrics, d) for d in Dimension}
    overall = "fail" if any(v["verdict"] in ("fail", "not covered") for v in dims.values()) else (
        "borderline" if any(v["verdict"] == "borderline" for v in dims.values()) else "pass")
    return {"overall": overall, "dimensions": dims}

Two decisions in that function are worth stating explicitly, because they are the ones that get argued.

“Not covered” is a failure, not a gap. A dimension with no metric assigned to it is a dimension nobody is checking, and a scorecard that reports it as neutral invites a release to ship with an entire class unexamined. Treating it as a failure means adding a dimension forces somebody to add a metric for it.

The worst verdict wins, and the driver is named. A reviewer reading “spatial: fail (driver: Moran’s I)” knows what to look at. One reading “spatial: 0.62” does not.

A realism scorecard with per-dimension verdicts and a tolerance sensitivity row Five dimension rows fill the card. Structural passes, driven by geometry validity. Distributional passes, driven by the Wasserstein distance. Spatial is borderline, driven by Moran's I, which sits within a few per cent of its tolerance. Relational passes, driven by the rank-correlation check. Integrity passes, driven by the duplicate rate. Each row shows the driving metric's value against its tolerance, so a reviewer can see how much margin there is without opening anything. The overall verdict is borderline, taken as the worst dimension rather than an average. Underneath, a sensitivity row gives the overall verdict at eighty per cent, one hundred per cent and one hundred and twenty-five per cent of the declared tolerances: it fails at eighty, is borderline at the declared value, and passes at a hundred and twenty-five. That row is what tells a consumer with tighter requirements than the contract's that this release would not satisfy them, which is information no single verdict carries. One verdict per dimension, the driver named, and how much margin there is structural pass driver: geometry validity 0 invalid / 0 tolerated distributional pass driver: Wasserstein W₁ 0.031 vs 0.050 spatial borderline driver: Moran's I 0.618 vs 0.610 floor relational pass driver: rank correlation Δ 0.024 vs 0.060 integrity pass driver: duplicate rate 0.0002 vs 0.0010 overall: borderline — the worst dimension, never an average SENSITIVITY ×0.80 tolerances → fail ×1.00 declared → borderline ×1.25 tolerances → pass
The scorecard as a reviewer sees it: five dimensions, one verdict each, and the driving metric named — with the detail available underneath rather than instead.

Fix, Part Two: Publish the Sensitivity, Not Just the Verdict

A borderline pass is the case the scorecard exists to surface, and a verdict alone does not tell a consumer what to do with it. The addition that makes it actionable is a small sensitivity table: what the verdict would be at nearby tolerances.

python
def sensitivity(metrics: list[Metric], factors=(0.8, 1.0, 1.25)) -> dict:
    """How the overall verdict moves if every tolerance were tightened or loosened."""
    out = {}
    for f in factors:
        scaled = [Metric(m.name, m.dimension, m.value, m.tolerance * f, m.higher_is_better)
                  for m in metrics]
        out[f"×{f}"] = scorecard(scaled)["overall"]
    return out

A release that passes at every factor is robustly fine. One that passes at the declared tolerance and fails at eighty per cent of it is telling a consumer with tighter requirements exactly what they need to know, and telling the producer that the tolerance is doing more work than it should.

Verification Step: Gate the Scorecard, Not the Metrics

python
def test_every_dimension_is_covered(metrics):
    covered = {m.dimension for m in metrics}
    missing = sorted(d.value for d in Dimension if d not in covered)
    assert not missing, f"no metric assigned to: {missing}"


def test_no_metric_has_two_dimensions(metric_registry):
    dupes = [name for name, dims in metric_registry.items() if len(dims) > 1]
    assert not dupes, f"metrics in more than one dimension: {dupes}"


def test_release_passes_the_scorecard(metrics):
    card = scorecard(metrics)
    failing = {k: v["driver"] for k, v in card["dimensions"].items()
               if v["verdict"] in ("fail", "not covered")}
    assert not failing, failing


def test_scorecard_is_recorded_with_the_release(manifest):
    assert "scorecard" in manifest["validation"], (
        "the verdict must ship with the artifact — a scorecard in a CI log is not evidence"
    )

The last test is the one that turns the scorecard from a build step into an artifact. A verdict that lives only in a CI log disappears when the log rotates, and the release it justified stays in production for years afterwards.

Edge Cases & Gotchas

A dimension nobody downstream uses. Tempting to drop, and worth resisting. A dimension no current consumer uses is one the next consumer will, and dropping it means the release history has a gap exactly where somebody later wants a comparison. Keep the metric, and if the tolerance is genuinely irrelevant, widen it deliberately and record why.

Widened tolerance versus explicit waiver for a known failing metric Two cards. Widening the tolerance makes the release pass immediately and costs nothing at the time. A later reviewer cannot tell a widened tolerance from one that was always that wide, so the fact that something is known-broken is erased. It does not expire, so it silently protects every future release too. And when the underlying cause is fixed, nothing prompts anybody to tighten it back, so the gate stays permanently weaker than intended. Recording an explicit waiver leaves the tolerance alone and attaches a record: the metric, the measured value, the reason, the owner and an expiry. A later reviewer sees exactly what was accepted and why. It expires, so it has to be renewed deliberately rather than forgotten. And when the cause is fixed the waiver simply lapses and the gate resumes at full strength with no action needed. A footer states the general rule this is an instance of: a decision recorded as data can be reviewed, and a decision recorded as a changed parameter cannot be distinguished from a design choice. A decision recorded as data can be reviewed; one recorded as a parameter cannot widen the tolerance A LATER REVIEWER SEES a tolerance — indistinguishable from a design choice EXPIRES? no — it protects every future release too WHEN THE CAUSE IS FIXED nothing prompts a tightening; the gate stays weak record a waiver A LATER REVIEWER SEES the metric, the value, the reason, the owner EXPIRES? yes — renewal is deliberate, not accidental WHEN THE CAUSE IS FIXED it lapses and the gate resumes at full strength The same argument runs through this whole area: the seed registry, the privacy ledger, the contract change log and a metric waiver are all instances of recording a decision as data so that somebody else can check it later.
A waiver is a record with an expiry; a widened tolerance is indistinguishable from a design choice.

Metrics that are not independent. Several distributional metrics on the same attribute will move together, so a dimension containing five of them is not five times better covered than one containing one. Worth noticing when reading a green dimension: coverage is about the kinds of defect represented, not the count.

A failing metric that is known and accepted. It happens — a legacy tolerance, a known limitation. Record it as an explicit waiver with an expiry rather than by widening the tolerance, because a widened tolerance is indistinguishable from one that was always that wide.

The scorecard passing while a consumer complains. The dimension set is incomplete, and the complaint is the specification for the missing metric. That is the most useful outcome a complaint can have, and it is worth asking for the failing case rather than only the report.

Who the Scorecard Is For

The artifact has three readers and they want different things from it, which is worth designing for rather than discovering.

The release reviewer wants a verdict and, when it is not a clean pass, one sentence naming what to look at. They are reading it under time pressure and they will read the top of it only. That argues for the overall verdict and the five dimension rows being the whole of the first screen, with everything else below.

The consumer wants to know whether this release is suitable for what they do, which is a different question from whether it met the producer’s tolerances. The sensitivity row is written for them: a consumer whose requirements are tighter than the contract’s can see immediately that the release would not satisfy them, without having to reason about individual metrics.

The auditor, later wants to know what was checked and what was accepted, possibly years afterwards and possibly about a release that has since been withdrawn. They need the full detail and they need it attached to the artifact rather than to a build log. That argues for the scorecard being recorded in the manifest and retained with it, which costs a few kilobytes.

Designing for all three is mostly a matter of ordering rather than content: verdict first, then sensitivity, then the per-metric detail, and all of it in the manifest. A scorecard that is a rendered dashboard and nothing else serves the first reader and fails the other two.