Preserving Off-Network Movement When Snapping

A consumer reports that synthetic traces are unusually clean: every fix lands on a road, journeys begin and end exactly at junctions, and nothing ever wanders. Real traces do none of that, and a detector trained to tell the two apart will separate them on this alone.

Part of Map Matching & Network Snapping: where the other pages are about making traces matchable, this one is about the portion of real movement that should not match, and what happens when a pipeline pretends it does not exist.

Root Cause: Two Ways to Erase the Same Thing

Off-network movement is a large fraction of real telemetry. A vehicle journey begins in a car park or a driveway, ends in another, and often crosses a forecourt or a service yard in between. A pedestrian journey spends much of its time in buildings, plazas and paths no network extract carries. None of it is on a road, and all of it is real.

Two independent decisions erase it, and a pipeline usually makes both:

The generator never produces it. Journeys are routed edge to edge on the network graph, so they begin at a node and end at a node. Every fix is by construction on a segment, and the trace has no off-network portion to preserve.

The matcher is not allowed to refuse. Most matchers can leave a fix unmatched, and most integrations do not let them: a nullable segment identifier is inconvenient downstream, so the pipeline takes the nearest candidate regardless of distance. A car park is then assigned to whichever road passes closest, and a real property of the data is replaced with a plausible-looking fabrication.

Off-network share of a journey, by distance and by elapsed time Two horizontal box-and-whisker summaries share a percentage axis. The upper one is the off-network share measured by distance: its median is a few per cent, and even its ninety-fifth percentile stays in the low single figures, which is why a pipeline that reasons about distance concludes off-network movement is negligible. The lower one is the same journeys measured by elapsed time, and its median is several times higher with a long right tail. The reason is in the speeds: the off-network portion happens where the agent is manoeuvring, parking or stationary, at a small fraction of the road speed, so a short distance occupies a long interval. Percentile markers give the median, the seventy-fifth and the ninety-fifth for both. The consequence stated underneath is that a pipeline discarding off-network movement discards a small fraction of the geometry and a large fraction of the dwell — which is precisely the part a trip-chaining or occupancy consumer depends on. A few per cent of the distance, and a fifth of the time by distance travelled median 3.0% p75 5.2% p95 10.9% by elapsed time median 12.3% p75 20.6% p95 39.9% 0% 7% 14% 21% 28% 35% 42% off-network share of the journey 5,000 simulated journeys, fixed seed. The off-network portion happens where the agent is manoeuvring or parking, at a fraction of the road speed — so a pipeline that discards it loses a little geometry and most of the dwell.
Measured over 5,000 simulated journeys: the off-network share is small in distance and large in time, because it is concentrated at the start and end where the agent is slow or stationary.

The chart makes the second half of that clear, and it is the half people underestimate. Off-network movement is a few per cent of the distance travelled and a substantial share of the elapsed time, because it happens where the agent is manoeuvring, parking or stationary. A pipeline that discards it discards the part of the journey that dwell analytics cares most about.

Minimal Reproducer: Compare the Fix Distributions

python
import math


def distance_to_network(fix, network_index) -> float:
    seg = network_index.nearest(fix)
    return seg.distance(fix)


def offnetwork_profile(traces, network_index, threshold_m: float = 25.0) -> dict:
    dists = [distance_to_network(f, network_index) for t in traces for f in t["fixes"]]
    dists.sort()
    return {
        "median_m": dists[len(dists) // 2],
        "p95_m": dists[int(0.95 * len(dists))],
        "p99_m": dists[int(0.99 * len(dists))],
        "share_beyond_threshold": sum(1 for d in dists if d > threshold_m) / len(dists),
    }


print("real:     ", offnetwork_profile(real_traces, idx))
print("synthetic:", offnetwork_profile(synthetic_traces, idx))

The two profiles differ in a way that is trivially detectable. Real traces have a long right tail — a small share of fixes tens or hundreds of metres from any road — and synthetic traces routed edge to edge have almost none. The p99_m field alone separates them.

Fix: Generate It, Flag It, and Let the Matcher Say No

1 — Give every journey an off-network head and tail

python
def with_access_legs(rng, route_xy: list, origin: dict, destination: dict) -> list:
    """Prepend and append the movement between a real place and the network."""
    head = access_leg(rng, origin["location"], route_xy[0], origin["kind"])
    tail = access_leg(rng, route_xy[-1], destination["location"], destination["kind"])
    return head + route_xy + tail


def access_leg(rng, a, b, kind: str) -> list:
    """A short, slow, meandering leg — a car park aisle, a driveway, a forecourt."""
    steps = {"car_park": 14, "driveway": 4, "forecourt": 8, "kerbside": 1}[kind]
    out = []
    for i in range(steps):
        t = (i + 1) / steps
        # a deliberate lateral wander: manoeuvring is not a straight line
        wander = rng.normal(0, 3.5) * math.sin(math.pi * t)
        out.append((a[0] + (b[0] - a[0]) * t + wander,
                    a[1] + (b[1] - a[1]) * t + wander * 0.6))
    return out

The access leg is short in distance and long in time, and generating it as a slow meander rather than a straight dash is what makes the speed profile plausible: a vehicle crossing a car park is doing five kilometres an hour, not fifty.

2 — Flag every fix with its expected matchability

python
from dataclasses import dataclass


@dataclass(frozen=True)
class Fix:
    t: float
    x: float
    y: float
    on_network: bool          # what the generator knows
    context: str              # "road" | "car_park" | "driveway" | "pedestrian_area"

on_network is the generator’s own knowledge and it is the thing that makes validation possible: the matcher’s decision to leave a fix unmatched can be scored against whether it should have been.

3 — Configure the matcher to refuse, and keep the refusals

python
def match_with_refusal(fixes, graph, max_emission_m: float = 40.0):
    """A matcher that may return None for a fix, and a caller that keeps it."""
    result = viterbi(fixes, graph, max_emission_m=max_emission_m)
    return [
        {"fix": f, "segment": seg, "matched": seg is not None}
        for f, seg in zip(fixes, result.segments)
    ]

max_emission_m is the parameter that makes refusal possible: beyond it, no candidate is admissible and the fix is unmatched. Setting it to infinity — which is what “always take the nearest” means — is what turns a car park into a road.

Three matcher refusal policies scored on coverage and on both error kinds Three scores group the horizontal axis, each with three bars. Coverage is the share of fixes that received a segment identifier at all. A matcher that never refuses scores a perfect hundred per cent on it, which is the number that gets reported and the reason the configuration survives. The second score is the share of off-network fixes that were nonetheless given a segment: the never-refuses matcher scores a hundred per cent here too, meaning every car park, driveway and pedestrian area in the release has been assigned to whichever road happened to be nearest. The third is the share of genuinely on-network fixes that were wrongly refused, where the never-refuses matcher scores zero by construction. A tight emission ceiling inverts the picture: almost nothing off-network is placed, and a meaningful share of legitimate fixes is refused. A calibrated ceiling, tuned against the generator's own on-network flags, gets both error rates into single figures at the cost of a coverage number that is no longer a hundred. The note underneath states the trap the chart is about: coverage is the only one of the three that can be computed without ground truth, so it is the one that gets optimised, and optimising it guarantees the worst possible value of the second. Coverage is the only score you can compute without ground truth 0% 25% 50% 75% 100% per cent 100% 100% 0% 82% 9% 17% 94% 12% 4% coverage off-network fixes wrongly placed on-network fixes wrongly refused never refuses tight ceiling (15 m) calibrated ceiling (40 m) Coverage is computable without ground truth, so it is the score that gets optimised — and optimising it guarantees the worst possible value of the middle column. Scoring the other two requires the generator's own on-network flag, which is the argument for carrying it.
Scoring the matcher's refusals against the generator's own flags: a matcher that never refuses scores perfectly on coverage and badly on everything that matters.

Verification Step: Score the Refusals Both Ways

python
def score_refusals(matched: list[dict]) -> dict:
    tp = sum(1 for m in matched if m["matched"] and m["fix"].on_network)
    fp = sum(1 for m in matched if m["matched"] and not m["fix"].on_network)
    fn = sum(1 for m in matched if not m["matched"] and m["fix"].on_network)
    tn = sum(1 for m in matched if not m["matched"] and not m["fix"].on_network)
    return {
        "wrongly_placed": fp / max(fp + tn, 1),     # off-network fixes given a segment
        "wrongly_refused": fn / max(tp + fn, 1),    # on-network fixes left unmatched
        "coverage": (tp + fp) / len(matched),
    }


def test_offnetwork_fixes_are_not_placed(traces, graph):
    s = aggregate(score_refusals(match_with_refusal(t["fixes"], graph)) for t in traces)
    assert s["wrongly_placed"] < 0.15, s
    assert s["wrongly_refused"] < 0.05, s


def test_release_has_an_offnetwork_tail(traces, network_index):
    p = offnetwork_profile(traces, network_index)
    assert p["p99_m"] > 30.0, (
        f"p99 distance to network is {p['p99_m']:.0f} m — this release has no off-network "
        f"movement and is separable from real data on that alone"
    )

The last assertion is the unusual one, and it is the point of the page. Most validation checks that a synthetic release is not wrong; this one checks that it is not too clean, which is a failure mode with no natural alarm.

The Detectability Argument

There is a second reason to preserve off-network movement, separate from utility, and it is the one that usually decides the question once somebody raises it.

A synthetic release is often evaluated by an adversarial check: can a classifier distinguish synthetic traces from real ones? A release whose every fix lies on a road is separable from real data by a single feature — the distance from each fix to the nearest segment — computed without any model at all. The distribution of that distance in real telemetry has a long right tail, and a release routed edge to edge has essentially none, so the two are distinguishable at a glance and by any detector.

That matters beyond aesthetics. A release trivially separable from real data is a release whose statistical parity claims are hard to sustain: whatever the aggregate metrics say, a consumer can demonstrate in one line that the synthetic data has a property real data does not. Adding off-network legs closes the gap cheaply, and it closes it in a way that is also more useful, because the added movement is genuinely part of the journey.

The reverse trap exists too. A generator that adds off-network noise indiscriminately — scattering fixes away from the network everywhere rather than at the origins and destinations — produces the right marginal distribution and the wrong structure. A detector aggregating by position will see off-network fixes in the middle of motorways, which no real trace produces. The distinction is the same one that runs through this whole area: matching a marginal is not the same as reproducing a mechanism, and only the mechanism survives contact with a consumer who looks at where things are rather than how many.

Edge Cases & Gotchas

Indoor movement has no network at all. Pedestrian traces inside a building, a station or a shopping centre are entirely off-network by construction, and treating them as an error produces a release with no indoor movement in it. Flag the context and let the matcher refuse the whole stretch.

Storage cost and fix count for an eight-hour dwell against the reporting interval The horizontal axis is the reporting interval used during a dwell, from five seconds to five minutes, on a logarithmic scale. The falling curve, read against the left axis, is the storage cost of a single eight-hour dwell in kilobytes: at a five-second interval one stop occupies more than two hundred kilobytes, which for a population of agents dominates the artifact entirely, and it falls by more than an order of magnitude across the range. The other curve, read against the right axis, is the number of fixes the dwell still contains, with a dashed line marking the minimum a density-based detector needs before it will call the cluster a stop. Between five seconds and about two minutes the fix count stays comfortably above that minimum while the storage cost falls dramatically, which is the range worth using. Beyond it the count crosses the minimum and the dwell becomes undetectable, and the crossing point is marked. The note underneath states the rule the chart supports and the trap it warns against: reduce the interval during dwells rather than removing the fixes, because a dwell rendered as a gap has no cluster at all and is undetectable at any interval. Reduce the dwell interval; never render a dwell as a gap 5 10 20 30 60 120 300 0 100 200 300 reporting interval during a dwell (s, log scale) storage per 8-hour dwell (kB) 0 2,000 4,000 6,000 236 kB per stop size falls 24× and the dwell is still detectable storage per dwell (left) fixes in the dwell (right) An 8-hour dwell at 42 bytes per fix, against a detector needing 18 samples. Reduce the interval during dwells rather than removing the fixes: a dwell rendered as a gap has no cluster at all and is undetectable at any interval.
Reducing the dwell interval saves an order of magnitude of storage; removing the fixes entirely makes the dwell undetectable.

Car parks that are mapped. Some network extracts include parking aisles, in which case those fixes are legitimately matchable and the generator’s on_network flag depends on the network version — which is one more reason the release should carry the network version it was validated against.

Consumers who require a segment on every row. They exist, and their pipelines break on nulls. Give them a separate view with the nearest-segment fallback applied and a matched boolean beside it, rather than destroying the distinction in the primary release.

Speed checks fire on access legs. A five-kilometre-an-hour leg inside a fifty-kilometre-an-hour trace will trip a naive outlier check. That is the check being wrong rather than the data; condition the speed bounds on the context flag.