Language models can infer an unfamiliar labeling rule from examples in a prompt without changing their weights. A study by Stanford researchers Andrew Gao and William Gao traces an internal mechanism supporting this ability in controlled sentence-classification tasks: a memory built at label-token positions, with a repetition-related signal that helps switch retrieval on.
The paper, “A Circuit for In-Context Learning,” was posted to arXiv on November 3, 2026. Its peer-review status is unverified, and no independent replication is supplied. The main experiments examined Qwen3-4B-Base, SmolLM3-3B-Base and Llama-3.2-3B, primarily using two classes and up to 12 demonstrations. Tasks included sentiment, topic and English versus Spanish, with arbitrary or natural labels.
To test the computation, the researchers transferred internal activations between closely matched prompts, replaced attention-head outputs with their average values, and steered internal representations. Controls included substitutions with sentences of the same class and length, random head sets matched by size and layer, and random directions of equal strength. Selected head sets were evaluated on fresh prompt draws from the same pools.
The interventions support a key–value memory account. At each label token, the model retains the label’s identity as a value while middle-layer attention heads write the preceding sentence’s class as a key. Upper-layer “reader” heads compare the query’s class with those keys and retrieve matching labels. In Qwen, transferring label-token values in layers 21–29 carried 86% of the effect of exchanging labels. Changing one demonstration’s class shifted the six-example margin by −2.87 nats; transferring label-token keys in those layers carried −2.35 nats. The margin measures preference for the correct label over the alternative, rather than an accuracy percentage-point change.
A repetition-related signal mainly moved readers away from an attention sink at the prompt’s beginning, enabling retrieval. Removing 16 duplicate-token heads eliminated about half that signal; the source of the remainder was unresolved. A fitted softmax model predicted attention on held-out prompts with one example per class, with median errors of 0.07 in Qwen and 0.04 in Llama. It failed on longer, 18-example prompts, where errors rose to 0.15 and 0.13, respectively.
These results provide causal evidence about the tested internal computations, with clear limits. Artificial interventions can create states unlike ordinary inference. Evaluation prompts shared sentence pools and templates, limiting evidence for generalization and potentially making prompt-level bootstrap intervals optimistic. The authors used 4,000 bootstrap resamples, but uncertainty estimates are not visible for every reported effect. The study does not establish a universal mechanism for instruction following, long-form tasks or model reliability; broader claims in the abstract cannot all be assessed from the supplied experimental text.
Cover illustration generated with AI; conceptual, not research evidence.