
Multimodal AI models may benefit more from parameters reserved for processing images than from a separately pretrained vision encoder, according to Unbundling Multimodal Vision Encoding, an arXiv preprint dated November 12, 2026. The team, with NYU and Meta FAIR affiliations, reports that moving this dedicated capacity into early layers of a shared image–text backbone preserved visual question-answering performance. Peer review has not been verified.
The study separates three choices usually bundled into a vision encoder: knowledge acquired through separate pretraining, parameters used exclusively for vision, and processing images before they interact with text. Its main experiments trained components from scratch using common data sources and settings, comparing backbones with 1.7 billion and 4 billion parameters. A scaling study covered 0.6 billion to 6.4 billion parameters.
Training combined Nemotron-CC text with English MetaCLIP image–text data, using a 70:30 text-only/image–text mixture and regenerated captions for multimodal training. Comparisons matched total training floating-point operations across stages. That accounting excluded instruction tuning for evaluation: three epochs on Cambrian-7M at 448 × 448 resolution. Visual results averaged four category means across 16 benchmarks; language results averaged four category means across 12 benchmarks, rather than pooling every question.
At a 1.7-billion-parameter backbone and 2.6 × 10^21 training FLOPs, the jointly trained scratch encoder achieved 44.4% average visual question-answering performance, versus 39.0% without an encoder—a 5.4 percentage-point gap. Language averages were 49.7% and 50.7%, respectively. Adding the encoder’s 0.4 billion parameters to the shared encoder-free backbone yielded visual averages of 38.6–38.8%, leaving the gap unresolved, particularly for reading text and charts.
Where the dedicated parameters sat also mattered. Vision-only feed-forward branches in the first eight backbone layers scored 45.0%, compared with 44.4% for the scratch encoder; placing that capacity in late layers scored 35.1%. The authors describe their default early in-fusion design as comparable to the encoder across tested training budgets. Fitted scaling relationships suggested encoder-free models needed approximately three times the encoder’s compute to match visual scores and caption loss within the tested range. This is not a universal runtime or serving-cost ratio.
Separate pretraining offered no clear downstream advantage in this protocol. Even when its compute was excluded, longer MetaCLIP pretraining raised ImageNet recognition accuracy from 80.4% to 84.4% without improving average visual question answering.
The findings remain bounded by dense architectures, shared image sources and mostly 224 × 224 training images. They do not rule out benefits from released encoders trained on additional data. A higher-resolution patch-embedding variant also reportedly matched the encoder, qualifying broad claims against encoder-free approaches. The supplied evidence reports no independent seed replication, confidence intervals or formal equivalence tests: comparable scores are the authors’ characterization, not proof of statistical equivalence. Independent replication and broader architecture and data tests remain needed.
Cover illustration generated with AI; conceptual, not research evidence.