A simulation can miss the driving challenge it was designed to test: if an autonomous vehicle approaches an intersection slowly, a scripted oncoming car may pass before any conflict develops. A preprint posted to arXiv on November 6, 2026, proposes continuously adjusting surrounding traffic to preserve the intended interaction. Peer review has not been verified.
Researchers affiliated with the University of Toronto and McGill University describe a scenario orchestrator that monitors traffic and changes other vehicles’ timing, routes and assigned roles. It solves a constrained optimization problem at each step, seeking to maintain the target interaction while limiting interventions and preventing unrelated collisions. It receives the tested vehicle’s intended maneuvers, but not its subsequent detailed behavior.
The CARLA evaluation compares the orchestrator with ProSim, LASER and OSC2Runner. Six scenario families cover red-light violations, unprotected left turns, right turns, cut-ins, hard braking and overtaking. Six driving policies include three IDM variants, PlanT 2.0, SimLingo and TFv6. Together, the combinations form a stated 144-cell evaluation design, rather than a reported count of independent trials.
The evaluation separates whether a challenge actually occurred from how the driving policy handled it. Scenario completion requires three ordered scene conditions, with the same traffic actor fulfilling the interaction and its ending. After tuning on the basic IDM policy, the orchestrator averaged 3.0 out of 3 conditions in every scenario family across the held-out assertive and conservative IDM variants. Baseline averages ranged from 1.0 to 3.0. These figures measure scene completion, not percentages of successful trials.
For red-light and cut-in tests averaged together, the orchestrator matched or exceeded each baseline’s reported policy-failure score, tying for three policies. That score combines five categories: collision, near-collision, harsh swerving, off-road collision and failure to recover. It is not a crash probability or, by itself, a fair ranking of driving safety. More failures can also reflect a stronger adversary or a less avoidable challenge.
In one small comparison, the orchestrator generated collision scenarios against PlanT 2.0 in three of three randomly initialized left-turn runs; OSC2Runner did not within its allotted search budget. This supports targeted stress generation under those conditions, not a broader estimate of collision risk.
The evidence remains limited to selected simulations. Comprehensive rollout counts, seed handling, confidence intervals and statistical tests are not supplied, and differing tuning attempts may influence comparisons. Traffic realism and avoidability are not comprehensively validated. The work suggests a way to make simulated tests deliver their intended challenge more reliably; it does not establish real-world driving safety.
Source: Scenario Orchestration for Closed-loop Policy Evaluation of Autonomous Driving, arXiv:2611.09888, version 1.
Cover illustration generated with AI; conceptual, not research evidence.