We live in a multimodal world. We see, talk, act, and dream. Yet most LLMs still start with language pretraining. Why not train them natively with multimodal I/O from scratch? Because it’s SUPER HARD, adding modalities triggers training instability, design complexity, and often modal competition So what’s the path forward? Introducing: Towards Physics of Multimodal Pretraining (junlinhan.github.io/projects…) We unpack the underlying mechanics of multimodal pretraining across 4 aspects: Knowledge Flow, Modality Synergy, Early Unification, and Recipe.

A SINGLE encoder + decoder for all the 4D tasks! We release 🎯 D4RT (Dynamic 4D Reconstruction and Tracking). 📍 A simple, unified interface for 3D tracking, depth, and pose 🌟 SOTA results on 4D reconstruction & tracking 🚀 Up to 100x faster pose estimation than prior works

Introducing VGGT-Ω: scaling feed-forward reconstruction across static and dynamic scenes, and studying whether the learned geometric representations transfer beyond reconstruction.

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: szymanowiczs.github.io/lager… Code: github.com/facebookresearch/… Paper: arxiv.org/abs/2603.20176 Models: huggingface.co/collections/f… Work done with @jianyuan_wang @MinghaoChen23 Christian Rupprecht and Andrea Vedaldi

🚀📣 Excited to share our new paper: TRAVL — A Recipe for Making Video-Language Models Better Judges of Physics Implausibility! ⭐ GitHub: github.com/insait-institute/… 👨🏻‍💻 Project page: sam-motamed.github.io/projec… 🎥 TL;DR: Video generative models look increasingly realistic, but still break physics (teleportation, gravity violations, impossible deformations). If we want multimodal systems that reason about the physical world, we first need reliable judges of plausibility. Can VLMs become that plausibility judge? 🤔 We introduce: • 🍳 TRAVL — a drop-in trajectory-aware attention recipe that keeps VLM backbones frozen while making vision tokens richer in motion and scene detail. • 🗂️ TRAVL Training Dataset — 3,482 short videos with 19,708 physics-focused Q/A pairs (balanced real + implausible). • 🧩 ImplausiBench — a 300-video benchmark (150 real / 150 implausible) with matched first frames & adversarial MCQs to blunt language shortcuts; we report both Human and LLM-as-judge scores. 🏆 Result: LLaVA-NeXT + TRAVL achieves the best performance on the implausible split in our study — outperforming Gemini 2.5 Pro and GPT-4o! 💡 Why it matters: Current VLMs often miss physics violations because full-video encoding explodes token counts → training resorts to sparse frames + heavy pooling (losing motion cues), and most training data is only real footage. TRAVL fixes this by injecting spatial self-attention (within frames) + trajectory-guided temporal attention (across frames), so models track continuity, contact-before-motion, and no-teleport constraints — the stuff physics is made of. 🧰 Use it today: • Method: Add TRAVL on top of a frozen VLM to enrich motion & scene context without blowing up tokens. • Data: Fine-tune with our curated 3,482 videos / 19,708 QAs focused on physics plausibility. • Eval: Rigorously benchmark VLMs' physics understanding with ImplausiBench! Please leave us a ⭐if you find this work interesting! 👨🏻‍💻Project webpage: sam-motamed.github.io/projec… 🤗 Code and Datasets: github.com/insait-institute/… 📜 arXiv: arxiv.org/abs/2510.07550 👥 Authors: @sammtmd , @MinghaoChen23 , Luc Van Gool, Iro Laina @INSAITinstitute | @Oxford_VGG

Many Congratulations to @jianyuan_wang, @MinghaoChen23, @n_karaev, Andrea Vedaldi, Christian Rupprecht and @davnov134 for winning the Best Paper Award @CVPR for "VGGT: Visual Geometry Grounded Transformer" 🥇🎉 🙌🙌 #CVPR2025!!!!!!

⚡️ Introducing Bolt3D ⚡️ Bolt3D generates interactive 3D scenes in less than 7 seconds on a single GPU from one or more images. It features a latent diffusion model that *directly* generates 3D Gaussians of seen and unseen regions, without any test time optimization. 🧵👇 (1/9)

Introducing VGGT (CVPR'25), a feedforward Transformer that directly infers all key 3D attributes from one, a few, or hundreds of images, in seconds! No expensive optimization needed, yet delivers SOTA results for: ✅ Camera Pose Estimation ✅ Multi-view Depth Estimation ✅ Dense Point Cloud Reconstruction ✅ Point Tracking Project Page: vgg-t.github.io/ Code & Weights: github.com/facebookresearch/…