Researcher at @Waymo Research; ML/Embodied AI; PhD @Princeton ECE
NYC
- Tweets 48
- Following 287
- Followers 520
- Likes 117
I rarely repost content, but this is a step-change for robotics research. As an embodied AI researcher, I find this to be a truly special moment for the field
Introducing GEN-1.5, a one-shot learner.
It can learn new tasks in a few seconds. Show it what to do, and it generalizes.
This capability emerged from pretraining on physical data at scale, as a step towards our mission of building general intelligence for the physical world.
7
19
360
39,669
Jimmy (Tsung-Yen) Yang retweeted
📢 Accepted to TMLR, with reproducibility certification 🏅
v2 of our JEPA-WM study (arXiv:2512.24497) is out, with new data-scaling experiments, a Lipschitz analysis of multistep rollout training, and extended discussions.
Recap + what's new 👇
w/ @JimmyTYYang1, Jean Ponce, @AdrienBardes, @ylecun
My first PhD paper is out! 🎓
"What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?"
tl:dr: JEPA-WMs for robotics: learn dynamics on top of visual encoders, optimize actions towards goal 👇
w/ @JimmyTYYang1, Jean Ponce, @AdrienBardes, @ylecun
15
29
235
64,214
It was great to work with @BasileTerv987 on this project! The JEPA world model works well on unseen robot tasks. Thanks Basile for the amazing work!
My first PhD paper is out! 🎓
"What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?"
tl:dr: JEPA-WMs for robotics: learn dynamics on top of visual encoders, optimize actions towards goal 👇
w/ @JimmyTYYang1, Jean Ponce, @AdrienBardes, @ylecun
1
4
642
It is my pleasure to work with Raktim! A concrete example of how to use human videos to learn robot policies
Introducing DexWM: Dexterous Manipulation World Model
tl;dr: train on human videos; fine-grained actions; hand-consistency loss; DexWM+MPC -> zero-shot dexterous manipulation
w/ @_amirbar, @DavidJFan, @JimmyTYYang1, @GaoyueZhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, @ylecun
6
523
It was so much fun to work on this: llama3 + sim2real skills, all are open-sourced!
FAIR researchers (@AIatMeta) presented SegmentAnything and our robotics work at the White House correspondents’ weekend.
Llama3 + Sim2Real skills (trained with @ai_habitat) = a robot assistant
1
4
22
4,158
Jimmy (Tsung-Yen) Yang retweeted
FAIR researchers (@AIatMeta) presented SegmentAnything and our robotics work at the White House correspondents’ weekend.
Llama3 + Sim2Real skills (trained with @ai_habitat) = a robot assistant
Washingtonians delved into the world of artificial intelligence (AI) at the Washington AI Network’s inaugural weekend TGAIFriday Lunch for White House correspondents. trib.al/FwHF9Um
3
19
218
103,346
Very excited to see that Habitat 3.0 is finally out! Please also check out our paper here: ai.meta.com/static-resource/…
Announcing Habitat 3.0, simulating humanoid avatars and robots collaborating!
- Humanoid sim: diverse skinned avatars
- Human-in-the-loop control: mouse/keyboard or VR
- Tasks: social navigation and rearrangement
Over 1,000 steps per second on 1 GPU for large-scale learning!
5
18
2,839
Please check out our Spot Sim2Real repo! github.com/facebookresearch/…
It aims to reproduce the demos of ASC and LSC. We open-source the weights of the navigation, picking, and place skills, and the infrastructure to use Large Language Models (LLMs) to interact with those skills!
Habitat stable version v0.2.5 released!
-- 🤖 Hardware bridge for BD Spot
-- 📷 Improved PBR rendering for better synthetic scene visuals
-- 🤗 ai-habitat on HuggingFace
-- 🏠 Habitat Synthetic Scenes Dataset (HSSD)
and more...
github.com/facebookresearch/…
github.com/facebookresearch/…
1
3
15
2,214
Jimmy (Tsung-Yen) Yang retweeted
“We were very interested in testing out high-level reasoning and planning. We wanted to see...if the robot could figure out the steps it needed to take to solve the problem.” - Discover why @MetaAI chose Spot for their research. bit.ly/44ILlhG
8
35
175
70,729
This is our recent work building toward "reproducible" robotic research. Thanks @chris_j_paxton for leading such an amazing project!
The future of robot butlers starts with mobile manipulation.
We’re announcing the NeurIPS 2023 Open-Vocabulary Mobile Manipulation Challenge!
- Full robot stack ✅
- Parallel sim and real evaluation ✅
- No robot required ✅👀
aihabitat.org/challenge/2023…
2
10
1,352
our demo @CVPR!
FAIR and @gtcomputing researchers demoing our sim2real work on @BostonDynamics Spot at @MetaAI booth at #CVPR23.
1
2
17
1,754
Check our work at Meta AI #CVPR23 ! Spot is able to understand natural language instructions and complete the task for you!
We’ll also be showcasing a handful of demos on the showfloor. We brought Spotty to demo some of our exciting recent work towards general-purpose embodied AI agents.
More on this research ➡️ bit.ly/3N8LeVg
5/6
5
22
2,989
Jimmy (Tsung-Yen) Yang retweeted
We’ll also be showcasing a handful of demos on the showfloor. We brought Spotty to demo some of our exciting recent work towards general-purpose embodied AI agents.
More on this research ➡️ bit.ly/3N8LeVg
5/6
3
34
151
56,577
Come join us for an instruction-following robot assistant demo at #CVPR2023 Meta Booth by the FAIR Embodied AI team.
Give Spot a free-form natural language instruction for object fetching.
Details: languageguidedskillcoordinat…
🧵
6
51
251
105,332
We will be running the LSC demo at two locations at #CVPR2023:
Meta AI booth: Tuesday to Thursday 10:00 AM to 6:00PM.
CVPR Demo track booth: Thursday 10:00 AM to 6:00PM
Details: languageguidedskillcoordinat…
Come join us!
1
1
10
3,954
This is the joint work of my amazing colleague: Sergio Arnaud, Kavit Shah, @naokiyokoyama0 , Alex Clegg, @joannetruong , Eric Undersander, Oleksandr Maksymets, @sehoonha , Mrinal Kalakrishnan, Roozbeh Mottaghi, @DhruvBatraDB , and Akshara Rai
1
5
596