Lower pretraining loss did not reliably identify the language models that adapted best in a comparison of Muon and AdamW. Researchers at Carnegie Mellon University report that Muon offered a better balance between downstream learning and retention at small pretraining data budgets, while AdamW overtook it at larger budgets. Their arXiv preprint was published November 2, 2026; peer review is not verified, and no external replication was supplied.
The team compared OLMo-style models with 30 million to 600 million parameters, pretrained on 150 million to 154 billion DCLM tokens. Nearly 600 pretraining runs and 29,000 fine-tuning runs covered GSM8K, StackMathQA, Alpaca and MusicPile. Optimizers determine how model parameters change during training: AdamW adapts updates coordinate by coordinate, while Muon orthogonalizes matrix updates.
The evaluation paired held-out downstream loss with held-out DCLM loss after fine-tuning, using the latter to assess retention of pretraining performance. Sweeps of training configurations produced Pareto frontiers: the best observed combinations of the two losses. The researchers compared both checkpoints selected for lowest pretraining loss and the best frontiers across explored pretraining and fine-tuning configurations.
For 60-million-parameter models fine-tuned on StackMathQA, Muon dominated at 2.4 billion pretraining tokens, but larger tested budgets favored AdamW. Muon still achieved lower pretraining loss when configurations were selected for that objective. The authors report the reversal under both selection approaches and across the additional datasets. On StackMathQA, the crossover fell from 320 tokens per parameter for 30-million-parameter models to 10 for 600-million-parameter models. These are ratios rather than absolute token budgets, and the crossover depended on the downstream dataset.
To investigate the shift, the authors introduced the margin-to-spread ratio: the correct answer’s raw output score, or logit, minus the average incorrect score, divided by the standard deviation of incorrect scores. A larger ratio indicates greater separation relative to that spread. Its median over correctly predicted held-out DCLM tokens favored Muon at small budgets and AdamW at large ones. This analysis conditions on successful predictions, and its crossover did not necessarily coincide with the fine-tuning crossover.
A separate mathematical analysis used synthetic Gaussian key–value facts in a linear associative memory. After one update from zero initialization, full-batch gradient descent had a larger margin-to-spread ratio than exact-orthogonalization Muon at sufficiently large data budgets. Longer-training simulations across three seeds showed a corresponding reversal in resistance to output perturbations. These findings support a proposed explanation; they do not establish a causal mechanism in language models or directly prove the result for AdamW.
The evidence has practical limits. Loss measurements do not directly establish task accuracy, factual reliability or retention of particular knowledge. The frontiers approximate only explored configurations. Fine-tuning coverage was asymmetric: AdamW-pretrained models used AdamW, while Muon-pretrained models used both optimizers. The appendix comparison cited to justify that choice is unavailable in the supplied text, which ends before the detailed methods and results appendices.
No confidence intervals or language-model seed counts were supplied. Fine-tuning runs sharing a pretrained checkpoint are dependent, so the run count does not represent independent replications. Held-out evaluation was reported, but a separate final test set untouched by configuration selection was not established. The findings support evaluating adaptability alongside pretraining loss within the tested settings; they do not establish optimizer superiority for larger production models, other architectures or other adaptation methods.