Fixing Phantom Stops at Traffic Signals

A stop detector run over a synthetic trace reports far more stops than the generator placed, and they cluster at junctions. The trace is correct, the detector is correct, and the release is about to be rejected by a consumer counting activities.

Part of Stop Detection & Dwell-Time Modelling: this page is the most common phantom, why the instinct to suppress it is wrong, and what to do instead.

Root Cause: They Are Not Phantoms

The first thing to establish is that a signal wait is a real stop. The agent genuinely stopped, for a genuinely measurable duration, at a genuinely identifiable place. A detector reporting it is not malfunctioning.

The problem is one of category. A consumer counting activity stops — trips, visits, dwell analytics — does not want signal waits in the count, and a consumer studying congestion wants nothing else. The same detected stop is signal or noise depending entirely on who is asking, and a release that reports one undifferentiated list of stops is unusable to both.

There are three ways this goes wrong, in increasing order of how much damage they do:

  • Suppressing the wait in the generator. The traces no longer contain signal stops at all, and are trivially distinguishable from real traces: real vehicle traces are full of them.
  • Filtering them out after detection with a duration threshold. This works until it does not, because signal waits and short activity stops — a delivery drop, a passenger set-down — overlap in duration, so a threshold that removes the waits removes real activities too.
  • Leaving them in unlabelled. The consumer applies their own threshold, gets a different answer from the one the release was validated against, and reports the release as wrong.
Signal-wait and short-activity duration distributions, with the trade at three thresholds Duration in seconds runs along the horizontal axis from zero to seven minutes; the vertical axis is the number of stops. Two histograms overlap. The signal-wait population, in the accent colour, is concentrated in the first ninety seconds and is bounded by the signal cycle. The short-activity population, in the primary colour, is a log-normal centred around two and a half minutes with substantial mass below ninety seconds — deliveries, set-downs, brief errands. Between about thirty and one hundred and twenty seconds the two overlap heavily. Three vertical lines mark candidate duration thresholds, and beside each is the trade it offers: the share of real activities it discards and the share of signal waits it fails to remove. None of the three is acceptable — a threshold low enough to keep the activities keeps most of the waits, and one high enough to remove the waits discards a large fraction of the activities. The conclusion drawn underneath is that no scalar threshold separates overlapping distributions, so the category has to be carried as a label on the data rather than inferred from the duration. No threshold separates them, because the distributions overlap 0 60 120 180 240 300 360 420 0 1,000 2,000 3,000 stop duration (s) stops 60s → loses 7% of activities, keeps 0% of waits 120s → loses 37% of activities, keeps 0% of waits 180s → loses 63% of activities, keeps 0% of waits signal waits (4,960) short activity stops (3,040) 8,000 simulated stops, 90 s cycle with 40 s green plus congestion, fixed seed. No scalar threshold separates overlapping distributions — the category has to be carried as a label on the data rather than inferred from the duration.
Measured over 8,000 simulated stops: signal waits and short activity stops overlap substantially in duration, so no threshold separates them cleanly.

Minimal Reproducer: Show That No Threshold Works

python
import math


def simulate_stops(rng, n=8000):
    """Two populations that a duration threshold has to separate — and cannot."""
    signals = [min(cycle, rng.exponential(1 / 0.055))
               for cycle in (90.0,) for _ in range(int(n * 0.62))]
    short_activities = [math.exp(math.log(150) + rng.normal(0, 0.62))
                        for _ in range(n - len(signals))]
    return signals, short_activities


signals, activities = simulate_stops(rng)
for threshold in (30, 60, 90, 120, 180):
    kept_activity = sum(1 for a in activities if a >= threshold) / len(activities)
    kept_signal = sum(1 for s in signals if s >= threshold) / len(signals)
    print(f"{threshold:>4}s  activities kept {kept_activity:5.1%}   "
          f"signals still present {kept_signal:5.1%}")

Every row is a bad trade. A threshold low enough to keep the real short activities keeps most of the signal waits; one high enough to remove the signal waits removes a third of the activities. The two distributions overlap, and no scalar threshold separates overlapping distributions.

Fix: Generate Them Deliberately, and Label Them

The correct fix has nothing to do with thresholds. It is to make the category a property of the data rather than a property of the duration.

1 — Generate signal waits from the network, not from a distribution

If the simulation has signalised junctions, the wait should emerge from them:

python
def signal_wait(rng, junction: dict) -> float:
    """A uniform arrival within the cycle, waiting out the remaining red."""
    cycle = junction["cycle_s"]
    green_share = junction["green_s"] / cycle
    phase = rng.uniform(0, cycle)
    if phase < junction["green_s"]:
        return 0.0                                    # arrived on green
    return cycle - phase                              # wait out the red

This produces the right shape for free — a spike at zero for arrivals on green and a roughly uniform spread across the red phase — and it produces it at the junctions that have signals, which no fitted duration distribution can do.

2 — Carry the cause on every generated stop

python
from dataclasses import dataclass


@dataclass(frozen=True)
class Stop:
    arrive: float
    depart: float
    location: tuple
    cause: str          # "activity" | "signal" | "congestion" | "give_way"
    activity: str | None

The cause field is the whole fix. It costs one column, it is known exactly at generation time and unrecoverable afterwards, and it lets every consumer filter to the category they want without guessing at a threshold.

3 — Score the two categories separately, and publish both numbers

python
def score(intended: list[Stop], recovered: list[dict], contract: dict) -> dict:
    activity = [s for s in intended if s.cause == "activity"]
    network = [s for s in intended if s.cause != "activity"]
    matched = match_stops(intended, recovered)
    return {
        "activity_recall": recall_of(activity, recovered),
        "network_recall": recall_of(network, recovered),
        "unexplained_rate": unexplained(recovered, intended),   # the true phantoms
        "detector": contract,
    }

The third number is the one that was hiding. Once signal waits are labelled, the unexplained rate — detected stops that correspond to no generated stop of any cause — falls to a few per cent, and what remains is genuinely a defect rather than a category dispute.

The same detected stops before and after carrying a cause label Two horizontal stacked bars represent an identical set of detected stops. The upper bar is how they appear without a cause label: a portion matches the generator's intended activity stops, and everything else — well over a third — is counted as a phantom, because nothing in the release says what it was. The lower bar is the same detections once every generated stop carries a cause. The activity share is unchanged. What was previously an undifferentiated phantom block resolves into signal waits, congestion waits and give-way pauses, each of which is a real stop the generator deliberately produced, leaving a small genuinely unexplained remainder. That remainder is the only part that is a defect. Callouts give the phantom rate under each interpretation, and the difference between them is entirely a matter of what the release recorded rather than of what it contains. The note underneath makes the operational point: the cause is known exactly at generation time, unrecoverable afterwards, and costs one column. The phantom rate collapses because most phantoms were never phantoms unlabelled 58% matches an intended activity 42% phantom — nothing says what it was phantom rate 42% labelled by cause 58% activity 24% signal wait 11% congestion give-way genuinely unexplained phantom rate 3% The cause is known exactly at generation time and unrecoverable afterwards. It costs one column, and it is the difference between a release that is 42% wrong and one that is 3% wrong.
The same detected stops, before and after labelling: the phantom rate collapses because most of the "phantoms" were real stops with a cause the release never recorded.

Verification Step: Gate Each Category Against Its Own Threshold

python
def test_activity_stops_are_recovered(traces, contract):
    r = aggregate(score(t["intended"], detect_stops(t["fixes"], **contract), contract)
                  for t in traces)
    assert r["activity_recall"] >= 0.92, r


def test_unexplained_rate_is_small(traces, contract):
    r = aggregate(score(t["intended"], detect_stops(t["fixes"], **contract), contract)
                  for t in traces)
    assert r["unexplained_rate"] <= 0.04, (
        f"{r['unexplained_rate']:.1%} of detected stops match nothing generated — "
        f"this is a real defect, not a category dispute"
    )


def test_signal_waits_are_present_at_all(traces):
    """A trace with no signal waits is not a realistic vehicle trace."""
    waits = [s for t in traces for s in t["intended"] if s.cause == "signal"]
    per_journey = len(waits) / len(traces)
    assert per_journey > 1.5, f"only {per_journey:.1f} signal waits per journey"

The last assertion is the one that catches an over-correction. A team that has been burned by phantom stops often removes signal waits entirely, and the resulting traces fail a realism check on a property nobody was watching.

What the Cause Vocabulary Should Contain

A cause field is only useful if its values are stable and shared, so it is worth deciding the vocabulary once rather than letting it accrete.

Four values cover the great majority of vehicle traces. Activity is a stop the agent’s schedule intended, and it is the only category most consumers want. Signal is a wait at a controlled junction, derived from the phase rather than drawn. Congestion is a wait caused by the traffic state on a link rather than at a node, which matters because it is the one that is not at a junction and therefore breaks any rule keyed on junction proximity. Give-way covers uncontrolled junction delays, roundabout entries and pedestrian crossings, which are shorter than signal waits and occur at different places.

Pedestrian traces need two more. Crossing is a wait at a controlled or uncontrolled crossing, and access covers doorways, lifts and barriers, which produce short stops in places no road network has.

Two design rules keep the field useful. It should be single-valued and exhaustive: every generated stop gets exactly one cause, and there is a value for “other” so that nothing is silently uncategorised. And it should be generator knowledge only — never inferred after the fact, because an inferred cause is a guess with the authority of a data field, and a consumer cannot tell the difference.

The field also gives release notes something concrete to say. “Activity-stop recall 94 per cent, unexplained rate 3 per cent, against detector radius 35 m and minimum duration 180 s” is a sentence a consumer can act on. “Stop detection was validated” is not.

Edge Cases & Gotchas

Congestion produces waits that are not at junctions. Modelling only signals leaves a second population of network stops distributed along links rather than at nodes. If the simulation has a traffic state, use it; if it does not, generating congestion waits from a link-level probability is better than pretending they do not happen.

Three sources for network waits, by input, placement and failure Three cards. Drawing from a fitted duration distribution needs only observed durations; it can place a wait anywhere along the route, because it has no notion of where waits occur, and what it gets wrong is exactly that — waits appear at random points on links rather than at junctions, which is detectable by anybody who aggregates stops by location. Sampling a signal phase needs the junctions and their cycle and green times; it places waits only at signalised junctions, which is correct, and it produces the right shape for free — a spike at zero for arrivals on green and a spread across the red phase. What it gets wrong is everything that is not a signal: give-way delays, queueing behind a turning vehicle, congestion between junctions. Deriving from a traffic state needs a link-level speed or density field; it places waits wherever the state says movement is slow, which covers both junction and link causes, and what it gets wrong is that it needs a traffic model most pipelines do not have. A footer records the pragmatic middle ground: signal phase at junctions plus a link-level congestion probability covers most of the distribution without a full traffic simulation. Only the last two can put a wait where waits actually happen fitted duration distribution NEEDS observed durations PLACES A WAIT anywhere along the route GETS WRONG waits appear mid-link, not at junctions — visible to anyone aggregating by location signal phase sampling NEEDS junctions, cycle and green times PLACES A WAIT at signalised junctions only GETS WRONG misses give-way, queueing and between-junction congestion derived from a traffic state NEEDS a link-level speed or density field PLACES A WAIT wherever movement is slow GETS WRONG needs a traffic model most pipelines do not have The pragmatic middle: signal phase at junctions, plus a link-level congestion probability. That covers most of the distribution and most of the placement without a traffic simulation.
x

Signal waits and short deliveries occur at the same places. A delivery vehicle stopping at a junction is genuinely ambiguous, and the label is the generator’s own knowledge rather than an inference. This is the one case where the ground truth is strictly better than anything a detector could recover, which is the argument for shipping it.

Consumers who ignore the label. Some will. The release note should state the activity-stop recall and the unexplained rate as the headline numbers, so a consumer who filters on duration at least starts from a documented baseline rather than a surprise.

Pedestrian traces have the same problem with a different cause. Crossings, doorways, and waiting for a lift produce the same short-stop population. The cause vocabulary should be extended rather than the pattern abandoned.