Calibrating Agent Populations Against Observed Counts

The simulation is tuned until the count at the main cordon matches the sensor. Every other counter in the network is then wrong, several of them by more than a factor of two, and raising or lowering the agent count moves them all in the same direction without fixing any of them.

Part of Agent-Based Mobility Simulation: the population size is the parameter everybody tunes first, and it is usually the wrong parameter — the problem is almost always the distribution across origins rather than the total.

Root Cause: One Number Cannot Calibrate a Spatial Distribution

A single cordon count is one constraint. An agent population has as many degrees of freedom as it has origin-destination pairs, and fitting one constraint to a model with hundreds of free parameters leaves the model determined almost entirely by its priors.

Scaling the total agent count is the crudest possible response: it multiplies every flow by the same factor, so a network where half the counters read high and half read low cannot be improved by any scale factor at all. The best achievable scaling puts the error in the middle and leaves both halves wrong.

There is a second, quieter problem underneath. Sensor counts are not counts of agents; they are counts of detections, and detection is imperfect. An inductive loop misses closely spaced vehicles, a camera misses occluded pedestrians, and a Bluetooth counter sees only devices that are on and discoverable. Calibrating a population against uncorrected detections builds the sensor’s blind spots into the synthetic population, permanently.

Per-counter error under the best global scale factor Fourteen counters run along the horizontal axis. The vertical axis is the percentage by which the modelled count differs from the corrected observed count, after applying the single multiplicative factor that best fits the whole network. Bars above the line are counters where the model still reads high; bars below are counters where it still reads low. Roughly half fall on each side, several by more than thirty percent, and one by more than double. This is the best achievable result for any scaling: multiplying every flow by a constant moves all fourteen bars in the same direction at once, so reducing an error above the line necessarily increases one below it. The note underneath draws the conclusion — the problem is the distribution of the population across origins and destinations, not its size, and no amount of tuning the total agent count addresses it. Every counter moves together — so half of them are always wrong -60% -30% +0% +30% +60% counter modelled minus observed (%) 6 reading high 8 reading low 14 counters, the scale factor chosen to minimise median absolute error, fixed seed. Scaling multiplies every flow by the same number, so it moves all fourteen bars together: closing a gap above the line opens one below it. The parameter that needs fitting is the distribution across origin-destination pairs, and the total agent count is the wrong knob entirely.
Counter-by-counter error under the best possible global scale factor: no single multiplier fixes a distribution problem.

Prerequisite Check: Correct the Counts Before Fitting to Them

python
def corrected_count(raw: int, sensor: dict) -> dict:
    """Detections → estimated true count, with the uncertainty that implies."""
    rate = sensor["detection_rate"]              # measured, not assumed
    est = raw / rate
    # binomial variance on the detection process, propagated through the division
    var = raw * (1 - rate) / (rate * rate)
    return {"estimate": est, "sd": var ** 0.5, "rate": rate}


def usable_counters(counters, min_rate=0.6, max_rel_sd=0.15) -> list:
    """Drop counters too unreliable to constrain anything."""
    out = []
    for c in counters:
        corr = corrected_count(c["raw"], c["sensor"])
        if corr["rate"] >= min_rate and corr["sd"] / corr["estimate"] <= max_rel_sd:
            out.append({**c, **corr})
    return out

Dropping counters is the part people resist, and it is the part that improves the fit most. A counter with a detection rate of 0.3 and no independent estimate of that rate contributes noise weighted as if it were signal, and one such counter can distort a whole fit — because the optimiser will happily distort the flows across half the network to satisfy it.

Fix: Fit the Distribution, Not the Total

1 — Start from a prior origin-destination matrix

python
def gravity_prior(zones, impedance) -> dict:
    """A defensible starting matrix from zone attributes and travel cost."""
    return {
        (o.id, d.id): o.production * d.attraction * math.exp(-impedance[o.id, d.id] / 12.0)
        for o in zones for d in zones if o.id != d.id
    }

The prior is doing real work here, and it should be chosen from data that is independent of the counters — zone populations, employment, land use. A prior derived from the counters themselves makes the subsequent fit circular and the goodness of fit meaningless.

2 — Fit with iterative proportional fitting against every counter

python
def ipf(prior: dict, counters, assignment, rounds: int = 40, tol: float = 0.01) -> dict:
    """Scale OD flows so every counter's modelled count matches its corrected estimate."""
    flows = dict(prior)
    for _ in range(rounds):
        worst = 0.0
        for c in counters:
            using = assignment[c["id"]]                  # OD pairs whose route crosses c
            modelled = sum(flows[od] for od in using)
            if modelled <= 0:
                continue
            factor = c["estimate"] / modelled
            worst = max(worst, abs(factor - 1))
            for od in using:
                flows[od] *= factor ** 0.5              # damped: pairs cross many counters
        if worst < tol:
            break
    return flows

The damping exponent matters. An OD pair crossing six counters gets six corrections per round, and applying each at full strength makes the process oscillate rather than converge. A square root is the usual compromise and is stable across a wide range of network densities.

3 — Sample agents from the fitted matrix, reproducibly

python
def instantiate(flows: dict, root_seed: int) -> list:
    """Agents drawn from the fitted matrix, one deterministic stream per OD pair."""
    agents = []
    for (o, d), rate in sorted(flows.items()):          # sorted: order fixes the seeds
        rng = seeded_rng(root_seed, f"od:{o}:{d}")
        for _ in range(poisson(rate, rng)):
            agents.append(Agent(origin=o, destination=d, depart=departure_time(rng)))
    return agents

Per-pair seeding rather than one global stream means adding a zone to the model does not reshuffle every other zone’s agents, which is what makes two calibration runs comparable.

Iterative proportional fitting under three damping exponents The horizontal axis is the fitting round, from one to twenty-two. The vertical axis is the worst relative error across all counters at the end of that round, on a scale where zero means every counter matches its corrected estimate. Three curves are drawn. The undamped fit, applying each counter's correction at full strength, drops fast for a round or two and then oscillates without settling, because every origin-destination pair crossing several counters receives several full corrections per round and overshoots each time. The over-damped fit descends smoothly but slowly and has not reached tolerance by the last round. The square-root exponent descends nearly as fast as the undamped fit initially and continues to a low error, crossing the tolerance line within a handful of rounds. A tolerance line is drawn across the chart. The note underneath explains why an intermediate exponent is stable across a wide range of network densities. How hard to apply each correction decides whether the fit settles 1 6 11 16 21 0.0 0.4 0.8 1.2 1.6 fitting round worst counter error tolerance undamped (exponent 1.0) square root (0.5) over-damped (0.15) 40 origin-destination pairs, 10 counters, each pair crossing roughly a quarter of them, fixed seed. An intermediate exponent works because a pair crossing k counters receives k corrections per round: applying each at full strength multiplies the intended adjustment by k and overshoots, while the square root keeps the compounded step near the right size across a wide range of network densities.
Counter error across fitting rounds under three damping choices: undamped oscillates, over-damped crawls, and the square root converges.

Verification Step: Hold Out Counters and Check Them

python
def test_fitted_counters_match(flows, fit_counters, assignment, tol=0.05):
    for c in fit_counters:
        modelled = sum(flows[od] for od in assignment[c["id"]])
        assert abs(modelled - c["estimate"]) / c["estimate"] < tol


def test_held_out_counters_match(flows, holdout, assignment, tol=0.15):
    """The real test: counters the fit never saw."""
    errs = [abs(sum(flows[od] for od in assignment[c["id"]]) - c["estimate"]) / c["estimate"]
            for c in holdout]
    assert median(errs) < tol, f"median held-out error {median(errs):.3f}"


def test_flows_stay_near_the_prior(flows, prior, max_log_ratio=1.6):
    """A fit that moves a flow by 5x has fitted noise, not signal."""
    for od, v in flows.items():
        assert abs(math.log(max(v, 1e-9) / max(prior[od], 1e-9))) < max_log_ratio, od


def test_population_is_reproducible(flows, root_seed):
    a = instantiate(flows, root_seed)
    b = instantiate(flows, root_seed)
    assert [x.signature() for x in a] == [x.signature() for x in b]

The held-out test is the only one that distinguishes calibration from curve fitting, and the third is the one that catches a fit which achieved its match by making implausible flows. Both are routinely omitted, and a fit that passes only the first test is indistinguishable from one that memorised the counters.

Fitted and held-out counter error against fit size The horizontal axis is the number of counters included in the fit, from two up to thirty, on a scale that spreads the small values. The vertical axis is median relative error. The lower curve is error on the counters the fit was given: it is low everywhere and gets slightly lower, which is expected and carries almost no information. The upper curve is error on counters held out of the fit entirely. With only two counters in the fit the held-out error is very high — the fit matched what it saw and learned nothing generalisable — and it falls steeply as more counters are added before flattening out. The vertical gap between the two curves at any point is the overfitting at that fit size. A marker shows where the gap first falls below the acceptance threshold. The note underneath states the practical rule: a fit reported only on its own counters is uninformative at every fit size, and the held-out curve is the one that says whether the population is calibrated or memorised. The gap between the curves is the overfitting 2 8 16 24 30 0.00 0.10 0.20 0.30 0.40 counters used in the fit median relative error acceptance threshold held-out clears at 12 held-out counters counters used in the fit The lower curve carries almost no information: a fit matches what it was given at every fit size, including sizes where it has learned nothing. Reporting only that number is how a memorised fit gets certified. The held-out curve is the one that answers whether the population generalises off the instrumented corridors, which is where most of the synthetic agents actually travel.
Fitted versus held-out error as more counters enter the fit: the gap between the two curves is the overfitting.

Edge Cases & Gotchas

Counters on the same corridor are not independent constraints. Three sensors along one road constrain one flow three times and give the optimiser a false sense of coverage. Weight by corridor rather than by counter, or the fit will be dominated by whichever road happens to be best instrumented.

Time-of-day matters more than the total. A population matched on daily totals with the wrong temporal profile produces peak flows that are badly wrong, and peak flows are what most consumers actually use. Fit by period, using the same time-varying transition structure the routing model needs anyway.

Route assignment is part of the model being fitted. The assignment map above assumes routing is known. If the routing model is also being calibrated, fitting flows against counters while the routes move produces a fit that tracks the routing changes, and the two have to be alternated rather than optimised together.

Sensor drift over the observation window. A detection rate measured once and applied to a year of counts propagates any drift straight into the flows. Where the rate is re-measured periodically, use the period-matched rate; where it is not, the width of the resulting uncertainty belongs in the release notes.

Reporting a Calibration Somebody Else Can Trust

A calibrated population is a modelling claim, and it needs the same treatment as any other claim in the release: enough recorded detail that a reader who was not present can judge it.

The counters used, and the counters not used. Which sensors entered the fit, which were held out, and which were dropped for low detection rates — with the rates. A fit reported without its exclusions is not reproducible, and exclusions are exactly where a fit can be quietly improved.

The held-out error, as the headline number. The fitted error is the number people report and the number that carries no information. Leading with held-out median error, and giving the distribution rather than only the median, is what makes the calibration reviewable.

The prior, and how far the fit moved from it. A fit that multiplied some flows by five has told you more about the counters than about the population. Reporting the distribution of log-ratios between fitted and prior flows shows immediately whether the fit is a correction or a reconstruction.

The detection-rate assumptions, separately. These are the least certain inputs and the most consequential: a detection rate wrong by ten percent scales the whole population by ten percent, and it does so invisibly. Stating them explicitly lets a reader who knows the sensors better than the modeller does correct the result without re-running anything.

The period the calibration applies to. Populations calibrated on a term-time weekday do not describe August, and a release that does not say which period it represents will be used for both.