[3/9] Towards Simplicity. We also wanted to simplify the architecture. Using separate encoders for visual understanding vs. generation adds unnecessary complexity. We prove that you only need ONE encoder to excel at both. Just use RAE (e.g. SigLIP 2 or WebSSL). Raw pixel with x-pred is a promising future direction.

[4/9] The “Modality Tax” myth. Does adding vision destroy your language modeling? Mostly not. We find that raw video is very compatible with language modeling. The performance drop often associated with multimodal training actually comes from the distributional gap of image captions, not from visual inputs themselves! Adding multimodal data just slightly hurts your OOD text generalization, but with large benefits for tasks such as visual understanding and generation, and even world modeling.

[5/9] World Models <-> Multimodal Models. World models don’t need specialized architectures nor large domain-specific data! By formatting actions directly as text tokens and training with general multimodal data, the model can naturally learn tasks such as navigation, and be used for MPC planning. No latent actions, no specialized action heads. We even show that the model can qualitatively generalize OOD using free-form language like “get out of the shadow!”, unlocking a nearly infinite set of actions, unlike existing works such as Genie. This is an insight that I'm personally very excited about :)

[6/9] Multimodal MoE Works! It’s well-known that MoE provides effective yet efficient scaling for LLMs. But what about unified multimodal models? We show that multimodal MoE benefits greatly from higher granularity and sparsity, especially when paired with semantic visual representations such as RAE. We also see that MoE naturally learns modality specialization from the data, flexibly allocating capacity per input, unlike hard-coded designs such as modality-specific FFN.

[7/9] MoE Bridges the Vision / Language Scaling Asymmetry. We provide the first IsoFLOP analysis for both vision and language within the same model. Our analysis shows that vision and language scale differently; vision is more data hungry while language is more parameter hungry - it’s hard to be compute optimal for both. But MoE saves the day! MoE bridges the gap so that the data requirements of vision and language become nearly identical. This means that MoE is not just a way to efficiently scale, but also an architectural necessity for building unified multimodal models.

[8/9] What's Next? Ultimately, we hope this work pushes the community towards unifying multimodal and world models, to build systems that truly understand and reason about the physical world. We aren’t quite there, but this is our first step towards this vision. @ylecun

[9/9] On a personal note: I grew a lot from this project. More challenging than the technical hurdles was navigating organizational dynamics, advocating the research vision, and securing the compute resources. A huge thanks to the incredible team that made this a reality, to FAIR for the support, and especially to @TongPetersb + @__JohnNguyen__. We started this journey together and stuck by each other’s side through all the highs and lows. This work builds upon all of our collective experiences in and long-term beliefs about visual representations + multimodal modeling. I see this as just the first step in a long-term research agenda. I couldn't ask for a better partnership!