Robotics Research Intern @NvidiaAI; Robot Learning PhD student @UMRobotics; undergrad @imperialcollege
- Tweets 162
- Following 376
- Followers 337
- Likes 2,013
Pinned Tweet
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
8
33
257
66,673
Jayjun Lee retweeted
Our ecosystem partner @nvidia and Michigan Robotics recently published ADEPT, their latest breakthrough in accelerating robotic dexterity. We're pleased to see the Flexiv #Rizon 4s adaptive robot paired with the Sharpa dexterity hand for real-world validation. Check the paper👇
1
3
8
634
Jayjun Lee retweeted
Super interesting to see that the #SharpaWave hands beat parallel grippers on peg insertion: 5-10s vs 20-70s per trial. Less need for precise orientation at grasp, faster in-hand rotation, faster overall. And also that it confirms the trend we're seeing with #TacMap (co-authored with Sharpa) and T-Rex (from @Dantong_Niu
): tactile policies hit 8/10 success vs 3/10 vision-only. Wave's per-fingertip sensors simulate well in IsaacLab, rich contact data in training.
Our new work with Jayjun explores sim-first approach to pre-training and post-training. There is a lot to squeeze out of simulations and we still don't know the ceiling and this work is a step in that direction. Ideally, you want to work on problems that are compute bound or alternatively figure out a way to turn your problems into compute bound problems and training in simulations at scale is a compute bound problem with lots of opportunities.
This work also explores training visuo-tactile policies in simulation. @SharpaRobotics hand has 5 tactile sensors at the finger-tips and we simulate them in IsaacLab to provide rich interactive data in the loop.
We transferred policies zero-shot on two different arm-hand combinations and importantly we didn't tune any hyperparams associated with RL training or reward. Many tasks can be defined as goal reaching problems where the robot has to reach for the object, lift, transport and bring it to a desired goal and this is embodiment agnostic and in simulations, you can both generate and verify goals.
1
3
21
3,597
Jayjun Lee retweeted
Our new work with Jayjun explores sim-first approach to pre-training and post-training. There is a lot to squeeze out of simulations and we still don't know the ceiling and this work is a step in that direction. Ideally, you want to work on problems that are compute bound or alternatively figure out a way to turn your problems into compute bound problems and training in simulations at scale is a compute bound problem with lots of opportunities.
This work also explores training visuo-tactile policies in simulation. @SharpaRobotics hand has 5 tactile sensors at the finger-tips and we simulate them in IsaacLab to provide rich interactive data in the loop.
We transferred policies zero-shot on two different arm-hand combinations and importantly we didn't tune any hyperparams associated with RL training or reward. Many tasks can be defined as goal reaching problems where the robot has to reach for the object, lift, transport and bring it to a desired goal and this is embodiment agnostic and in simulations, you can both generate and verify goals.
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
2
7
68
11,898
Jayjun Lee retweeted
Fantastic work by @jayjunleee and the @NVIDIARobotics Dex research group. Always great to see new foundational sim2real work leveraging Isaac Lab!
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
1
1
7
1,285
Jayjun Lee retweeted
Dexterity should be a prior, not a per-task cost!
ADEPT learns reach-grasp-reorient-transport once in sim, then post-trains specialists that deploy zero-shot on real hands from pixels and touch: 3B steps per new task vs ~9B from scratch.
Great work @jayjunleee & @NVIDIAAI Dex
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
5
41
5,552
Jayjun Lee retweeted
It’s incredible how close we r coming to zero shot transfer on hardware!!
Hoping one day we can zero shot transfer for any task or morphology. This is one step towards that!
Great job guys!
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
1
1
240
Jayjun Lee retweeted
Amazing! Sim-to-real will be critical for dexterous hands I am sure of it.
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
3
3
37
8,011
Jayjun Lee retweeted
Incredible work! DextrahRGB proved visuals-action could work, so this is the obvious progression to tactile and a meaningful one!
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
1
3
575
Jayjun Lee retweeted
Always exciting to see sim2real RL working 🔥
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
2
24
4,604
Jayjun Lee retweeted
Simulation will enable super-human dexterity.
Our robots learn 23/29-DoF hand/arm policies from scratch, entirely in simulation, and transfer 0-shot to real.
Pre-training on a reorientation task accelerates dexterity learning on other tasks.
This is just the beginning, so many exciting techniques and results to share soon.
Great work by @jayjunleee and Dex team at NVIDIA
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
2
23
2,700
Jayjun Lee retweeted
Awesome new work from the dexterity team!
1. It's not always trivial to collect data that matches your morphology.
2. It's even harder to collect recovery data.
Object reposing is a great way for learning a generative prior --> fine-tune to handle new and unseen tasks.
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
1
17
2,774
Jayjun Lee retweeted
ADEPT: Accelerating Dexterity via Pre-Training.
What if dexterous robots could learn foundational manipulation skills once, and then reuse them across many different tasks?
ADEPT pretrains a dexterous RL policy on a generic object reposing task, learning useful motor primitives that can later be reused as a prior for downstream tasks.
The goal: make learning complex dexterous behaviors more efficient and transferable.
4
6
59
3,586
What if robot hands could acquire dexterity once, then reuse it to learn downstream tasks much faster?
Introducing ADEPT: a pre-training & post-training paradigm using RL entirely in sim, then zero-shot deploying visuo-tactile policies in the real world.
🔗adept-dexterity.github.io with Dex team @NVIDIAAI
8
33
257
66,673
The resulting policies solve complex manipulation tasks in just 5–10s with a multi-fingered hand, 2–14× faster than parallel jaw grippers, without external fixtures, pose trackers, or scripted task decomposition.
We also find that touch can signifcantly improve the real-world success rates for sim2real.
1
2
12
824
This work is done during my internship at NVIDIA and this project wouldn't have been possible without my collaborators at the Dex Team and UMich: Jessica Yin, Asif Rana, @nmblauch, Sam Mady, @mostlymohak, @NimaFazeli7, @robot_trainer, Karl Van Wyk, @ankurhandos.
Paper: arxiv.org/abs/2608.19182
Webpage: adept-dexterity.github.io
@NVIDIARobotics @UMRobotics
1
11
711