High accuracy in medical physics calculations does not necessarily make a language model a reliable checker of someone else’s work. That is a central finding of MedPhysBench, a benchmark described by Omar Nusrat in an arXiv preprint posted on November 3, 2026. Peer review has not been verified, and the evaluation does not measure patient outcomes or establish readiness for clinical deployment.
The distinction matters because errors in beam calibration or treatment calculations can systematically affect delivered radiation doses. MedPhysBench separates producing a numerical answer from checking a colleague’s worksheet or final result.
Ten seeded generators create problems across radiation therapy, diagnostic imaging, nuclear medicine and radiation protection, at three difficulty levels. Reference answers come from protocol-based code. A deterministic grader converts units and applies tolerance bands ranging from 0.5% to 2%, depending on the task. Worksheet scoring requires identifying the first wrong step; comparisons of error detection across checking formats use the verdict alone.
The author evaluated 12 models through Prime Inference in October 2026. The held-out test contained 600 calculation items, 300 worksheet checks and 300 result-only checks. GPT-6 Astra and Claude Fable 5.1 received a prespecified stratified half. Models answered without tools, using provider-default settings, with one scored response per item. Checking formats shared the same underlying cases, split equally between correct calculations and calculations with one planted error.
Reported calculation accuracy ranged from 33.3% to 100.0%. Four models scored 99.3–100.0%, and adjusted pairwise tests did not distinguish those four. However, review of their nine answers graded incorrect found five model errors and four answers that followed NCRP Report 151 where the benchmark misstated a rule or left an assumption open. Across six models, 12 rejected calculation answers followed that report.
Below the leading group, weaknesses often concentrated in the hardest tasks. Claude Sonnet 5.5 omitted an inverse-square correction in 19 of 20 isocentric monitor-unit calculations. A follow-up using high reasoning effort reduced that count to 14 of 20, leaving a systematic failure that its overall score could obscure.
Five models detected at least 96.7% of planted errors with no more than 4% false alarms in both checking formats. The planted errors were mostly large. Among the other seven models, showing only the final result significantly increased false alarms for four, under Holm-adjusted tests.
An exploratory follow-up examined the omitted inverse-square correction in 20 public cases. Three leading models caught every planted worksheet error, while three weaker models accepted 16–18 faulty worksheets. Claude Opus 5.5 accepted six wrong final results despite applying the correction in all 20 of its own calculations. This targeted test was selected after the main results and cannot estimate general failure rates.
The limitations extend beyond model behavior. These are synthetic, text-only tasks with supplied inputs, not independent clinical checks. Final-answer grading can miss protocol violations within tolerance or errors that cancel each other. The reference implementations shared a specification developed with an AI coding assistant; their agreement does not prove conformity to the governing documents.
Balanced test cases also do not establish how useful alarms would be where real errors are rare. The findings support evaluating calculation and checking separately on specific tasks, while leaving clinical responsibility with qualified medical physicists.
Cover illustration generated with AI; conceptual, not research evidence.