An independent reimplementation of practical Direct Nash Optimization, or DNO-Prct, improved two language models’ scores on an instruction-following benchmark, according to a Synopsys-affiliated research team. The arXiv preprint was published on November 3, 2026; peer review has not been verified. Its experiments were conducted between July and December 2024.
DNO approaches preference learning as a two-player game, seeking a strategy that holds up against competing strategies. The practical version repeatedly generates fresh training examples from the model being updated. Using UltraFeedback’s 63,967 prompts, the researchers trained Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1 for three iterations. Each round combined student-generated answers with a teacher response, model-assigned quality ratings, selection of sufficiently separated preference pairs, replay of some earlier pairs and a Direct Preference Optimization-style update.
Evaluation used 805 AlpacaEval 2.0 prompts, with GPT-4o judging answers against reference responses from GPT-4o and GPT-4-0613. Against GPT-4o, Llama’s length-controlled win rate rose from 31.71% to 45.32%, a gain of 13.61 percentage points. Mistral’s rose from 6.77% to 18.44%, gaining 11.67 points. Length control statistically adjusts for response length; it does not independently verify answer quality.
The tuned Llama 8B scored above Llama 3.1 70B Instruct on all four reported win rates. Their length-controlled scores against GPT-4o were 45.32% and 41.40%. This is a comparison within the study’s benchmark protocol: the two models were each evaluated against common opponents, rather than directly against each other. The custom judge, references and decoding settings also prevent direct comparison with public leaderboard scores.
Replacing GPT-4o’s rating role with larger, same-family open-weight models retained 62% of Llama’s and 82% of Mistral’s length-controlled gains against GPT-4o. That supports using an open-weight rating model, but does not establish a wholly proprietary-model-free pipeline: the supplied source passages do not establish which model supplied teacher responses in those runs. Separate component-removal experiments reported roughly 6–10-point losses in length-controlled scores when teacher responses or earlier-pair replay were removed, or pair selection was restricted to teacher-preferred responses. Different experimental configurations limit comparisons across these tests.
Several uncertainties constrain the findings. Every training configuration ran once, leaving training variability unknown, and comparisons on shared evaluation prompts lacked paired uncertainty analysis. GPT-4o supplied teacher answers and ratings in the main runs while also judging evaluation and serving as an opponent, creating scope for shared-model preference bias. Llama’s average response length increased from 2,237 to 3,786 characters; length adjustment may not fully resolve the influence of longer responses.
Baselines used different response sources and training schedules, and there was no matched-teacher supervised-training control. The study therefore cannot isolate DNO’s optimization method as the cause of the gains. It supports benchmark-specific improvement under the tested conditions, without establishing broader human preference alignment or convergence to the theoretical Nash equilibrium.
Cover illustration generated with AI; conceptual, not research evidence.