A simple example of physical prompt steerability: same environment, different prompts, different behaviors. Thanks @JagdeepBhatia8 for the suggestion. Read more about GEN-1.5 in our blog post in the comments below.

We've reduced the time it takes to go from physical prompt → robot behavior. The faster anyone can teach a robot to do something new, the easier it becomes to scale physical work. Read more about GEN-1.5 in our blog post in the comments below.

I'm usually wary about talk of "the ChatGPT moment for robots" because language is less complex than the physical world. After visiting @GeneralistAI, though, where I saw their robots perform impressive manipulation with just a prompt, I think you can see it happening in a limited way. (Thanks to @peteflorence and co for having me!) wired.com/story/generalist-a…

Good question thanks Ani. Here's a video that gives a sense of the answer here about the effect of prompting: - in the first part, the model is just babbling with no prompt in context - at the end, we add a prompt for taking money out of a pouch (generalizing to the wallet)

We started planning yesterday’s @generalistai GEN-1.5 announcement about 3 weeks ago. Two weeks ago, already feeling whiplash from new results, I asked ChatGPT to create this image to share in our internal meme channel.

I remember having a conversation with @ilyasut in the early days of @OpenAI (~2018) about how the pace of progress in AI research can be surprisingly fast. Overnight, things can go from feeling nearly impossible to just working. I felt it then, and I feel it now.

absolute GPT-3 moment for robotics holy moly

Ever since I started working on robot foundation models a handful of years ago, the broad ability to one-shot in-context learn has been the single most vivid goal in my mind along the long road ahead. It’s something @andyzengineer and I have especially been thinking together on for several years now, and something shared by the whole team at Generalist as a major goal. But even before anybody started talking about robot foundation models, too, the capability this enables, i.e. generalized one-shot learning, has been both incredibly concrete and elusive. In terminology that I think many other people have come up with as well, in grad school we used to talk about a “Ctrl-C-Ctrl-V” type of capability. See something, and have the robot do it too. This same type of idea inspired the name of the 1970 MIT copy demo (people.csail.mit.edu/bkph/ph…). The thing is, it sounds simple, but is incredibly hard since the world is never quite the same when it got copied and where you want to paste it. The real world can be hard to predict and is full of variation. To do this, you need strong generalization, and it needs to be acquired in a single example. Doing this over a wide range of tasks, especially for dexterous tasks, is hard mode. Lots of the components of the idea of making this all happen have been there for a long time. As an example, this 2017 NeurIPS paper “one-shot imitation learning” arxiv.org/pdf/1703.07326 has excellent vision, with ambition well beyond what was achievable at the time, and although it’s not referred to as “in-context learning” since it was pre-Transformer, it actually uses attention to condition on a single demonstration. And now, many things have happened since early 2017, including the broadly celebrated arrival of one/few-shot in-context learning in language models in 2020. This new model GEN-1.5 takes in everything we have built and learned over the past couple years at Generalist. It has been training for 8 months. It has taken an incredible amount of commitment and grit from the whole team to get here. The level to which this model has survived many surgeries has continued to surprise me. And its capabilities have continued to surprise as well. We found compositional generalization on Friday. We found sim2real prompting earlier last week. We filmed the contiguous uncut videos of live prompting yesterday. To be clear, the success rates are modest, and there’s still a long way to go. But now I have definitely seen a ~decade-long imagination come into the real world.

It feels like almost every day at this company there is another significant result! The speed of progress makes my job of documenting and communicating what we are doing a challenge, but it's also so much fun!

Our latest pre-training comes with zero-gradient / in-context learning! So crazy to witness and glad we can talk about it publicly now.

RT @felixwyw: At 10:06pm on Aug 3, I watched a robot do something I thought was years away. We were working on few-gradient learning with…

GPT3!

During a recent visit to Generalist AI, I watched a robotic arm improvise and use a banana as a tool. wired.com/story/generalist-a…

Introducing GEN-1.5, a one-shot learner. It can learn new tasks in a few seconds. Show it what to do, and it generalizes. This capability emerged from pretraining on physical data at scale, as a step towards our mission of building general intelligence for the physical world.

The models are getting better everyday. This is a pretty contact-rich task, and forces us to reckon with the fact that our policies not only need to learn the quirks of different robots, but also have to do it in a way that generalizes to the fine-grained subtleties of friction and forces e.g. so that objects don’t get wedged or stuck. Gave me goosebumps when I first saw these results internally. Something about it too, that just makes these models feel just a bit more “human” now in the way they move than they did before. Wild times.

What happens when you change the hand mid-task? We tested this by modifying the hands mid-rollout and letting the same model keep running. It perceives the new tool, and finds a new trajectory and contact strategy to complete the task. This works because training on mixed data forces the model to condition its behavior on the hand in front of it. This brings it closer to a general understanding of how shapes and contact surfaces interact with the physical world – and which actuation strategy should follow.

Each hand is a different sensorimotor interface by which GEN-1 experiences the physical world. Scaling pretraining across thousands of these interfaces teaches GEN-1 a universal physical commonsense that transfers to new hands and new ways to grasp, push, pull, twist, and more.

Sometime next year pairs of arms become reliable enough that they can do work on an assembly line or fulfillment center I think? In the same way that ~the most important trendline to have a view on is how many GW of accelerator capacity are deployed in 26/27/28/29, you should also have a view on how many pairs of arms are built and deployed

A year ago, I thought world models would be disruptive by sometime in the 2030s. It’s now likely that these models will have a huge impact THIS decade, and massively accelerate innovation and disinflation even by 2030. Generalist is awesome / bullish 🇺🇸

easy to miss detail and pretty neat: you can just modify the UMI style data collection devices to use different tools