👀 What if better visual reasoning isn't about more pixels or more vision encoders, but knowing WHICH visual representations to use? Excited to introduce Mixture of Layers (MoL), our work at #NeurIPS2026! 🎉 🧵 Here's how we make vision encoders work smarter:

「3年後にAGIが誕生すると分かっていながら，博士号を取り，ポスドクになり，大学教授になる計画を立てているんですか？」 Anthropicの研究者が，博士課程の学生たちに「あなたたちのキャリアプランはもう存在しない」と告げた講演が話題です。 拡散モデルの発明者でもあるJascha Sohl-Dickstein氏（Anthropic）の講演『Advice for a young investigator in the first and last days of the Anthropocene（人新世の最初と最後の日々における若手研究者へのアドバイス）』を紹介します。 youtube.com/watch?v=3LZN2UUx… 人類が主役だった地質年代（人新世）が，AIの台頭によって極めて短い期間で幕を閉じるかもしれない ——そんな前提のもと，提示されたデータは強烈です。 ■ 突きつけられた冷徹なファクト モデル訓練の計算量は過去10年で約1000万倍に増加し，人間の脳が生涯で行う総計算量に迫っている。AIが単独で完遂できるタスクの時間スケールは，数ヶ月ごとに倍増している。ここ7ヶ月ほど，モデルは数学者が長年解けなかった未解決の「Erdős（エルデシュ）問題」を解き始めている。サンフランシスコでは「長いタイムライン（懐疑派）」とは「AGIは10年以内に来る」を意味している。2030年の計算量は2025年の約1万倍に達する見込み。 線形グラフで見れば，現在の劇的な進歩すら「まだゼロと区別がつかない平坦な初期段階にいる」というのが彼の主張です。 ■ 学生たちへのシンプルで容赦ないアドバイス 毎日AIを使え。指数関数の上にいれば，遅れを取ることはない。彼自身はClaudeを「実行力は天才だが、研究のセンスはかなり悪い大学院生」のように毎日使い，自分ではもうコードを一切書いていない。 計算量がスケールするほど良くなるプロジェクトを選べ。それ以外は，完成する前に解かれてしまう サットンの『苦い教訓』の通り，スケールに味方されるテーマを選ばなければ，2年かけた研究も完成する頃には次世代モデルのプロンプト一発で追い抜かれる。 キャリアを「安定したもの」として計画するのをやめろ。安全な選択肢は，もはや安全を買ってくれない 社会全体で大規模な破壊が起きるなら，ベースラインのリスク自体が高い。従来の「手堅い選択肢」は，もはや安全を担保してくれず，ただ潜在的なリターンを犠牲にするだけになる。 そして，最も印象的な一言がこれ。 「書くことは考えることだ（Writing is thinking）。書くことを外注すれば，考えることも外注することになる。」 コーディングや実行は徹底的にAIへ委譲しつつ，問題設定（センス）と思考の核（書くこと）だけは決して手放すな。 AGIのタイムラインを本気で信じているなら，キャリアの計画も本気でそれに合わせろという，極めてストレートなメッセージの講演を紹介しました。 x.com/ecommartinez/status/21…

🚀 Introducing AdaExploreBench, a benchmark for evaluating proactive exploration and test-time adaptation in LLM agents. Can LLM agents proactively explore to understand and adapt to the environmental dynamics? Thanks to all collaborators! Paper: arxiv.org/abs/2610.25950

Ahead of launch, @robocurve evaluated Dream-1 independently. Result: a new Pareto frontier for robot control. 16x cheaper, 1.7x faster, and better performance (59 vs. 48 on RoboDojo-RC Tier 1) than the previous best alternative. LLMs can plan. Robots still need a cheap, fast, and general System 1 to act.

Introducing Embodied First Exam: One line of code to run coding agents in 1,090 embodied tasks across 31 simulation suites and 44 robots. Can your agent go far beyond pick-and-place: Puzzles with spatial constraints; Olympic sports with multi-robot collab; Real-time eval, where the world won't pause; Dynamic, agile, whole-body control; AutoResearch with sim2sim Try your own agent: harbor run -d embodied-first-exam -a <your_own_harness> -m <your_own_model>

Huge congratulations on the work from our lab! Really exciting to see this. I've been thinking about why embodied AI remains challenging, and where the next breakthroughs will come from. AI has repeatedly taught us: many seemingly architectural problems are indeed data problems. And this makes embodied intelligence so interesting. Data from physical world is hard to get. We cannot simply scrape billions robots interaction trajectories like web pages. But if we manage to have that embodied data, even models like astra originally built for language tasks can potentially solve embodied tasks. So I increasingly believe the next frontier of AI will be defined as much by where data is missing as by where model capabilities are missing. And this should not stop at robotics. Much of the world's intelligence exists as tacit knowledge: the hands of experienced technicians, the judgment of nurses, the intuition of farmers, and countless small decisions people make every day to keep the physical world running. Frontier models becomes more capable at embodied tasks, which we should celebrate. But a this doesn't mean that embodied intelligence is nearly solved, or that the remaining challenges will automatically disappear with the next model release. This raises a broader question: how much of the world's intelligence have we actually captured in a form AI can learn from? It's easy to forget how many people have never paid for AI, especially when we're surrounded by people who hit their coding agent usage limits every 5 hours. Some of the most interesting challenges lie in environments where collecting a single useful example is harder than training on a million existing ones. That's what makes embodied intelligence so interesting, and why I'm excited to see Embodied First Exam helping us understand what's still missing. Congratulations again to the team. Excited to see where this goes!

We are hiring PhD AI Research Interns to work on various projects around Multimodal LLMs for Summer 2027 (Reality Labs). Please reach out to me directly via email with your resume! metacareers.com/profile/job_…

Today, Dream Labs is out of stealth. In February 2026, while we were at NVIDIA, @scott_e_reed @youliangtan @GeYunhao and I started calling a new kind of robot model a World Action Model, one that learns how the world moves from video, then uses that to decide how to act. Today, we’re releasing Dream-1 through a public API that anyone in the world can use. When you rent a car you’ve never driven, you don’t learn to drive all over again. Within a few minutes, it feels like your own. Robots should work the same way: a good model should be able to pick up a new robot and get to work, without being retrained from scratch. That’s what we’ve spent the last 6 months on, and it’s what Dream-1 does. It controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. Generalization has been my North Star in robotics: first by distilling video world models into VLAs (DreamGen), then by learning to act from those world models directly (DreamZero), and now by transferring across robots (Dream-1). I’ve come to believe it’s the only direction that lasts. Pushing a narrow model from 90% to 99.9% on a single task is real work, but much of it gets washed away once a model that generalizes arrives. Since the start of the company we’ve stayed single-minded about generalization.That focus is what got Dream-1 to control robots it was never trained on, on tasks it was never shown. And it’s what makes a public API possible at all: a model that only does one task on one robot can’t serve millions of users. Dream-1 is still a research beta, and it will make mistakes. But it already drives cars it’s never seen. Generalization wins in the end, and the end is closer than it looks. Here’s Dream-1 taking the wheel of a robot it’s never seen. Full details 👇

User simulators are only as faithful as the user representations they are conditioned on. Missing details force simulators to fill gaps with assumptions, leading to generic behavior⚠️ Introducing UserRepGen-4B, which generates representations optimized for simulation fidelity.

Paper: arxiv.org/abs/2610.22518 Model: huggingface.co/microsoft/Use… This work was done during my internship @MSFTResearch and would not have been possible without my amazing collaborators and mentors: @PhilippeLaban, Michel Galley, @dilekhakkanitur, @ProfJenNeville, @JianfengGao0217.

Excited to share what we have been up to

October 19

Really cool to LeRobot Robotic harness with MolmoAct2 as the VLA with Qwen3-VL for planning. Try it out myself.

Spent years watching models ace a single image and completely whiff on anything that moves or changes angle. Today we’re fixing that. Try Glimpse now, our first visual thinking model with competitive frontier performance compared to Astra on key public visual reasoning benchmarks.

We are open-sourcing V1.0 of our agentic robotics framework, ASPIRE, with full real robot skill discovery and self improvement capabilities! On the day Astra arrived, our lab switched the backend model and saturated all the benchmarks to ~99% over night. And the frontier is already on the move: - Opus 5.5 slightly edges out Astra on our robot simulation bench. - Sonnet 5.5 hits the Pareto frontier of cost vs. robot capability, $12/task. We first introduced ASPIRE and /robot skill library in June, way before the big models could reliably do robotics. A rising frontier tide lifts all boats, but Astra was a 0->1 tsunami on many tasks. Now you can also feel the Physical Bitter Lesson at home like we did on that fateful day ;) links in thread

Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents? We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration. A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills. The results: • ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control. • +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro. • One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs. The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning. We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it! Project + sessions: robot-agent-exploration.com/ Paper: assets.robot-agent-explorati… GitHub: N Project fully led by @RunhanH and @xczhangxx who are absolutely awesome. How it works, experiments, and robot videos below 🧵 (1/N)

PingpongDuo Humanoid table tennis now has doubles, while I'm still single. proj: archon.tech/research/Pingpon… abs: arxiv.org/abs/2610.20579

🥳 Amazing time at #CSCW2026! 🇺🇸 Loved the inspiring talks, great conversations, and meeting amazing people! Excited to share two papers I presented! 👀 How can we design GenAI that accounts for human relationships and the people affected by its use? 👤 — 🤖 — 👤 A thread 👇

🤖 Sergey Levine’s #COLM2026 mic-drop: “ALL AI will be built on physical interaction.” Sergey @svlevine spotlighted our @saturdayrobotic #18 speaker @Guanming717 & @gen_instinct. 6 takeaways: MEMORY, π0.7, SARL, What matters? Big picture. @COLM_conf

In general, optimization dynamics for post-training is quite under-studied even today. In our subnetwork and SGD paper, while we tried to share some quite counter-intuitive findings, the bigger message was probably that this is worth studying. Some interesting papers if you are interested in the broad direction - (including @yufanzh_ 's paper below) arxiv.org/abs/2505.11711 arxiv.org/abs/2602.07729 arxiv.org/pdf/2607.19331 arxiv.org/abs/2511.08567 arxiv.org/abs/2609.36262 Some key takeaways from this dump - 1. on policy data causes sparse updates to the base policy 2. instead of looking at updates per individual weight, you can also look at how much each individual matrix moved via looking at their spectrum and UV subspaces, and surprise ;) they dont move much either (@zhu_hanqing666 👀) 3. Diagonal Fisher plays a big role in this "small movement" ( @yufanzh_ ) 4. The optimization landscape is non-stationary and momentum could hurt training (@lifan__yuan @pavanjayasinha ) ...... I recently gave a talk on this (calendars.illinois.edu/detai…). We are cooking some more fun stuff, and always happy to discuss ;)