>/BASH

AWS tests training-loop checks for silent GPU corruption

AWS reports that three checks can detect selected injected faults during distributed AI training and help identify suspect hardware. Tests on H100 GPUs show a trade-off between detection speed and training overhead, with important gaps in coverage.

AWS tests training-loop checks for silent GPU corruption

AWS has described a way to detect selected forms of silent data corruption inside distributed AI training jobs and connect the resulting alarms to node replacement on Amazon SageMaker HyperPod. Its November 11, 2026 technical blog reports software-injection tests, rather than an evaluation of naturally occurring hardware failures. Peer review and independent replication are not established by the supplied source.

Silent corruption occurs when hardware returns an incorrect but plausible number without reporting an error. In data-parallel training, averaging gradients across GPUs can spread that error to every model replica while their weights continue to agree. Memory and transport integrity checks do not establish that the arithmetic itself was correct.

The proposed PyTorch DistributedDataParallel workflow uses three complementary checks. It checks gradients for NaN, infinity or overflow before averaging, then checks the averaged gradient before applying an update. On scheduled audit steps, GPUs replay another node’s micro-batch and compare loss values bit for bit; deterministic computation also permits comparison of gradient fingerprints. After updates, fingerprints identify weight copies that differ from a strict majority. These compact signatures avoid exchanging entire tensors, but matching fingerprints are evidence of agreement, not proof.

AWS tested a 953-million-parameter model using four HyperPod instances with 32 NVIDIA H100 GPUs, and reports repeating every fault scenario on three instances. Injected faults affected forward calculations, backward gradients and stored weights. In one example, a mandatory weight check before checkpointing detected corruption and blocked the save. Disabling that safeguard allowed the corrupted copy to be saved; a replay audit detected the disagreement five steps later. The number of independent injection trials and uncertainty around detection rates were not reported.

Detection frequency carried a substantial cost. Over 1,000 consecutive steps per schedule, plain DDP averaged 157.4 milliseconds per step. Deterministic audits every 100 steps, with weight checks every 10 steps, averaged 173.3 milliseconds, a 10.1% increase. Auditing every 10 steps raised the average to 211.9 milliseconds, or 34.7% overhead. The less frequent schedule can allow up to 99 corrupted updates before detection, assuming the fault persists and affects the kernels being audited. Timing uncertainty was not reported.

Fault-free runs produced no alarms across 15,320 audit steps and 3,080 weight checks. AWS calculated an approximate 95% upper bound of one false alarm per 5,100 audit steps. That estimate depends on the tested workload and sampling assumptions; correlated steps limit its interpretation. A simpler gradient-norm outlier approach performed poorly: a threshold calibrated on 800 healthy steps produced 11 false alarms in the remaining 22,000 steps, while tested finite faults scored no higher than 6.9 median absolute deviations.

Important gaps remain. Finite errors inside gradient averaging can pass every check, and a one-time finite fault between audits can escape detection. Deterministic audits may also miss defects confined to default kernels. Replay requires reproducible data and computation, and reliable localization needs suitable placement across at least three nodes. Multiple faulty GPUs can cause incorrect attribution. Passing checks therefore does not certify a clean checkpoint.

For recovery, AWS recommends stopping the job, preventing new scheduling on suspect nodes, reviewing telemetry and resolving ambiguous alarms before replacement. Clean hardware metrics do not rule out silent corruption. Training should resume from a checkpoint preceding the suspect window, stored on shared storage because replacement destroys local volumes. AWS recommends human approval for production replacements until users establish precision on their own workloads. The evaluation does not quantify production recovery success or improvements in final model quality.

Cover illustration generated with AI; conceptual, not research evidence.

Original sources

View this story on Instagram