Success on familiar kinds of problems may conceal a fragile reasoning strategy, according to a controlled experiment by Ziqi Pang and Yu-Xiong Wang at the University of Illinois Urbana-Champaign. Their language model learned to work backward through synthetic string functions, then frequently switched to inefficient guessing when the tasks became harder. Adding a different kind of training problem helped preserve structured reasoning.
The research was posted to arXiv on November 2, 2026. It is a preprint; peer review is not verified, and independent replication is not supplied.
The researchers trained Qwen2.5-7B-Instruct using reinforcement learning with verifiable rewards, or RLVR, with GRPO and DAPO-style filtering. Each prompt supplied a function’s code and an output string, asking the model to find digit-string inputs that produced that output. A reference implementation checked the answers. The main training library contained 24 many-to-one functions: 18 used for training and six held out. Training contained 50,000 tasks, with 1,024 tasks per evaluation condition.
The baseline evaluation already used unseen functions, but matched the training regime of output lengths 4–6 and two requested solutions. Harder tests requested 3–6 solutions, increased output lengths to 6–10, combined those changes, or introduced functions with different input and output lengths. Reported accuracy allowed partial credit for correct, distinct answers and penalized excess submissions; it was not the percentage of tasks solved completely.
The model trained only on many-to-one inversion, called Mono, reached approximately 90% baseline accuracy. Structured derivation was identified in 85% of its baseline traces, but fell below 50% in the three harder out-of-distribution conditions. Accuracy stayed above 80% when only more answers were requested, while falling to 6% on length-changing functions. The authors report that many failures involved brute-force enumeration continuing until the token limit.
A second model, Mixed, used a 1:1 mixture of the original tasks and bijective inversion problems, which have a unique solution. It achieved similar baseline performance and substantially better accuracy on harder tests, while approximately doubling derivation use relative to Mono. Exact Mixed accuracy values are unavailable in the supplied text. The auxiliary tasks were not uniformly harder and asked for only one answer, suggesting that task difficulty or similarity alone did not explain the benefit.
The authors propose that derivation eliminates groups of possible answers more efficiently than testing candidates individually. Their theoretical account describes this as information gained per token and explains how rewards for final answers could indirectly favor useful reasoning steps. Reducing the response ceiling from 8,000 to 5,000 tokens and adding digit constraints also increased derivation use. These results support the account within this task suite, rather than establishing a universal mechanism.
More derivation did not always mean better answers. Raising the bijective-to-non-bijective ratio to 3:1 increased derivation use, but the 1:1 mixture was more accurate. Trace audits found execution errors even when the intended strategy was selected, highlighting the continuing need for task-specific skills.
The findings have substantial limits. The main evidence concerns one 7B model and synthetic functions; exploratory mathematics and agent experiments do not establish broad transfer to practical reasoning. Evaluation tasks share function families, so 1,024 tasks are not 1,024 independent tests of generalization across functions. Strategy labels came from Gemini-3.0-Flash, with only pure brute force classified as enumeration and everything else as derivation; referenced judge-validation details are absent from the supplied text.
The authors report repeated runs and controls for task diversity and difficulty, but detailed results, run counts and uncertainty estimates are unavailable, preventing assessment of the main accuracy differences’ statistical significance. Changed trace labels do not prove that strategy change alone caused the gains, and strong baseline performance does not exclude capacity limits on harder tasks. The theory also assumes binary terminal rewards and a large-group approximation, whereas the experiments award partial credit. The study offers a controlled warning: learning an effective strategy does not guarantee that a model will keep using it when the problem changes.