Native multimodal still allows a separate encoder

People often treat "native multimodal" as a single architecture choice: one shared stack that eats pixels, audio and text from the first layer. When DeepSeek-V4.1-Flash still ships an independent DeepSeek-ViT plus a two-layer MLP projector into a language backbone, and still joins vision and text already in pretraining, the reaction is surprise. Had every serious model already dropped separate encoders?

That surprise mixes three ideas that sound alike but answer different questions.

Joint pretraining means vision and language already meet in the early training stages, not only in a late fine-tune. End-to-end optimization means gradients can update the perception path and the language path together, rather than freezing a vision tower forever. A unified backbone means every modality is processed by the same low-level network from the first layer. Native multimodal talk often slides from the first two into the third, as if cancelling the encoder were the proof of seriousness. Under the joint-pretrain and end-to-end reading, a separate ViT plus projector still counts as native multimodal.

What we want from a multimodal system is a working picture of what is going on that can be updated when new evidence arrives. Call that a world state. Within a modular design, eyes, ears and a language core can keep separate wiring diagrams while still building that picture. They have to pass useful information, and the model has to keep revising its picture when the scene changes.

Language already carries a lot of the world

A common objection is that a pure text model arrives empty of spatial or physical information. Language is already a compressed record of space, physics, cause and effect, time, and human action. Math and code compress those relations further. A model that learns the relations behind those symbols can form part of an internal map of the world before any pixel arrives.

That claim has empirical support. An ICLR 2024 paper, Language Models Represent Space and Time, finds identifiable spatial and temporal structure inside language models trained only on text. Some of the scaffolding for world modeling already lives in the language route. Multimodal work then has to show what vision adds on top of that scaffolding.

Perception and reasoning can keep separate chassis

If the language backbone can hold and revise part of a world representation, a separate vision encoder and a separate audio encoder are enough architecture for many systems. The vision tower extracts spatial, texture and semantic cues. The audio tower extracts timing and acoustic structure. A projector or another cross-modal interface hands those cues to the backbone. The backbone reasons, predicts, and updates its judgment about the current environment.

A frozen encoder bolted on at the end sits outside this line. Cross-modal alignment, joint training and feedback still matter. The point is narrower. Perception has to take part in learning and reasoning. Under the joint-pretrain reading, each modality can keep its own network shape. If information moves, a modular stack can still build cross-modal understanding.

DeepSeek-V4.1-Flash sits in that modular line: independent DeepSeek-ViT, a two-layer MLP projector, a language backbone, and joint handling of vision and text from pretraining. Under the joint-pretrain and end-to-end reading, it already is multimodal in the sense that matters for training.

The bottleneck may be moving

Here is a hypothesis. As language backbones get better at reasoning and at holding a world picture, the hard part of multimodal systems may shift. Early work spent most of its pain on aligning modalities. The next pain may be whether perception keeps enough detail, and whether the model can keep updating its world state from new observations instead of clinging to an old language prior.

Take a cup on a table. The model sees it. Something else then occludes the cup. Recognition is the easy ask. The harder ask is whether the model still maintains where the cup is and what state it is in while it is out of view, and whether it revises that judgment when the cup reappears and the evidence has changed.

That splits into two problems.

On the input side: did the perception modules keep enough spatial, geometric, motion and semantic detail?

On the state side: can the model use those signals for a reliable update, or does an older text prior override them?

Under matched compute and data, the fair test is which design perceives more accurately, reasons more carefully, and updates state more honestly when the scene drifts.

Explicit scenes and latent scenes can both stay

Executable scene tools such as SVG, Blender and Three.js are useful for a different reason. They turn spatial relations and motion into structures a model can simulate, render, check and correct. Continuous latent methods such as VAEs or JEPA-style predictors remain useful beside them. Explicit programs are good for structured operations and external checks. Continuous latents are often better for high-dimensional perception and fast state prediction. The two can sit side by side.

A tool verifies a world hypothesis; the model still has to hold a reliable world model. A direction I like is internal prediction first, tools on demand for simulation and check, then feedback that revises the internal judgment. The tool is a test harness. The state lives in the model.

A proposed direction for state that can be audited

Updating a world state is a representation problem. In agent settings it is also a permission problem. Who is allowed to change the live record, and can a later step tell which version of the picture it depends on?

My open runtime project RealityGate sketches one possible answer. The model may propose an update. Only an executor holding a runtime-issued, single-use permit may change the recorded state. A request that only declares a capability stays unauthorized until the runtime issues that permit. A missing permission fails closed with a recorded reason. That is a privilege-separated way to treat world-state edits as events with evidence attached, rather than silent rewrites inside the model's private context.

What a world model is judged by

Native multimodal work has made real progress. Modular perception and unified low-level modeling will likely sit next to each other for a long time, and win on different budgets, modalities and tasks.

A world model, on this framing, is judged by stable representation, prediction, and revision under new evidence. Architecture unity is a separate design choice. The sharper questions are whether a model can build a stable world representation from perception, predict how the world will change, and revise its judgment when new evidence arrives. The goal is to let perception make the model's picture of the real world more accurate and more willing to update.

Scope and limits

The DeepSeek-V4.1-Flash architecture details above follow public descriptions of that release as I have seen them: independent DeepSeek-ViT, two-layer MLP projector, joint vision-text pretraining. I have not re-audited the weights myself. If a later technical report changes those details, the concrete example should be swapped; the separation between joint pretraining, end-to-end optimization and a unified backbone would still stand.

The bottleneck-shift claim is a hypothesis, not a measured result. The cup example is a thought experiment for state maintenance, not a benchmark score. The ICLR 2024 citation supports identifiable space and time structure in text-only models; it does not measure multimodal world-state update, and it leaves open how much vision still adds on top of the language scaffolding.

This essay argues for a scoped reading of "native multimodal" under joint pretraining and end-to-end optimization. It does not claim that a unified low-level backbone never helps, or that dropping every independent encoder never improves perception fidelity. Matched-budget comparisons that settle those questions are outside the evidence here.

RealityGate is a small runtime prototype. It has not been evaluated as a multimodal world-model stack, and this piece does not claim it has solved multimodal world modeling. I would revise the framing if matched-budget comparisons showed unified backbones systematically better at perception fidelity and state update, or if privilege-separated state edits turned out to block legitimate perception-driven revisions rather than only uncontrolled ones.