NYC
- Tweets 750
- Following 622
- Followers 2,361
- Likes 4,003
Pinned Tweet
New Paper: Human-like Autonomy Emerges from Self-Play and a Pinch of Human Data.
We trained self-play RL on 60 years of simulation on 1 GPU in ~15 hours. Regularizing with 30 minutes of demonstration data produces much more human-like driving policies!
8
40
350
77,910
Worst thing about this is we have no idea if GPT-6 was specifically post trained on Nethack or not. Zero transparency about the training data and environments. You can't independently evaluate the significance of anything without knowing this.
In all seriousness, this is a startling achievement for GPT-6 Astra. kenforthewin.github.io/blog/…
(This is GPT-6 Astra beating Nethack on its 3rd try. Nethack is the original roguelike and one of the most famously hard games of all time. I have played a lot, and I've never ascended)
14
19
303
27,589
You have to assume in-distribution until proven otherwise
Data and RL Envs have been the main advance of the last year
1
5
199
Daphne Cornelisse retweeted
Worst thing about this is we have no idea if GPT-6 was specifically post trained on Nethack or not. Zero transparency about the training data and environments. You can't independently evaluate the significance of anything without knowing this.
In all seriousness, this is a startling achievement for GPT-6 Astra. kenforthewin.github.io/blog/…
(This is GPT-6 Astra beating Nethack on its 3rd try. Nethack is the original roguelike and one of the most famously hard games of all time. I have played a lot, and I've never ascended)
14
19
303
27,589
How can we build a jellyfish that swims like a real jellyfish — through soft mechanics and embodied AI? JellyRobot explores the anatomy and answer with computed soft and hard structures and an RL policy, making the jellyfish can swim in the water like a Cambrian creature. sylvpengbm.github.io/JellyRo…
4
10
45
3,130
Daphne Cornelisse retweeted
I'm super bored of LLM-stylized papers/blogs. I think we all are?
So I asked Claude to explain its tells: web.mit.edu/phillipi/www/cla…
Let's avoid these! I'd much rather read *your* words, see *your* rickety figures, than face down yet another of Claude's anodized corporate veneers.
26
40
580
42,851
The details on the sweep software can be found here: puffer.ai/blog/protein/
We basically swept all hyperparameters, you can checkout the craftax and default.ini in the repo.
I haven't analyzed the winning hypers yet, but you can see them in the browser here puffer.ai/constellation.html -> click on craftax in the top left
1
46
Very cool thanks for sharing! Do you think uncertainty-based exploration really fixes any of this? My sense is that uncertainty-based exploration "overfits" to the environment's noise and/or learning algorithm, instead of doing the kind of "exploration" that we think of as humans. Most exploratory actions in current algos look like... noise. Even if the people working on this don't like to admit it (me included).
Humans clearly seem to do something different by designing clever experiments (and you point towards this when you talk about a human reasoning that ice works against fire). You can setup an LLM to try to do something similar, but idk how one would make this cheap enough to run at the scale of pufferlib.
None of this is super clear in my head. Just some thoughts from someone whose spent a long time thinking about exploration.
1
75
Good question! I agree, human exploration seems more deliberate than most existing uncertainty-based RL algorithms. They capture the high-level goal of exploration (maximizing some measure of surprise), but can be brittle because it is difficult to distinguish between relevant and irrelevant variance.
Still thinking on this as well, but here is a lecture with some of these topics in case it is helpful: drive.google.com/file/d/1sUh…
1
24
Very cool results and interesting write-up!
A couple of thoughts:
1. At the core, this seems like an insanely difficult behavior prediction problem due to the quirk you mention. Given the limited information (i.e., you can't observe bullets), your success largely depends on your ability to accurately model your partner's behavior and exploit it. I'd expect modeling the opponent to play a larger role here than in, say, Chess.
2. Are you explicitly considering "diversity" (e.g., action dist entropy) in maintaining your pool of checkpoints? As you mentioned, the key here is probably figuring out how to make the policy explore a wide strategy space, but probably also which cpts to store.
4
187
Daphne Cornelisse retweeted
i'd like to keep it low key and say this is no big deal, but in reality this has to be the coolest project i've worked on maybe ever, and there's a lot more to come!
9
10
81
14,468
Daphne Cornelisse retweeted
Joseph and team have been pursuing a very different approach to AI compared to the mainstream. They build really fast simulators + optimizers — so fast that you can do RL from scratch, even on hard domains. It’s worth checking out! Very cool results!
3
18
342
32,431
oooh, so many things! not too long ago, it really felt like LLMs had a lot of "passive knowledge" but struggled with precise low-level control. seeing that change has opened up sooo much to figure out!
as embodied agents: how they explore the state/action/behaviour space; what drives that exploration (curiosity, interestingness, etc); how they generalize to unseen levels/worlds; how they adapt their behaviour to in-context information; how they manage long contexts (compaction, etc); and how they move between high-level plans, temporally extended actions, and low-level reactive control.
most exciting direction imo is RL post-training. LLMs already have so much relevant knowledge, and we can give them even more through wikis/tutorials. how efficiently can we turn that knowledge into optimal policies? i think RL can be the mechanism that turns passive knowledge into optimal action with very little additional environment interaction. e.g LLMs know a lot about Atari games right? could an LLM learn to play Atari really well with, say, 50-100k env steps of RL post-training? i believe this has to be possible (standard budget for tabula-rasa RL in Atari is 200M env steps).
as autonomous researchers: how they orchestrate, supervise, and steer auto-research programmes to create knowledge and improve downstream agents/tasks. how they combine knowledge across more breadth than most humans can deal with, and whether they can use that breadth to push beyond the current frontier of human knowledge.
also, seeing Astra beat Craftax made me reflect much more deeply on the challenges of sequential decision-making (which you describe so well in your article!) and what tabula rasa RL would need to overcome them
1
2
77
Interesting! Appreciate the detailed response. :)
“e.g LLMs know a lot about Atari games right? could an LLM learn to play Atari really well with, say, 50-100k env steps of RL post-training?”
> One thing I am trying to figure out for myself is: say you successfully do this. What have we learned then? How can this lead to an insight beyond “we can now post train LLMs on Atari”. Not trying to be pessimistic, but we often don’t even know what these models are (not) trained on 😔
There is definitely potential, but I feel that it can be challenging to turn such experiments into generally usable insights. I’m looking forward to learning more about your findings though!
1
1
72
Not yet! But is interesting to try. My intuition is that starting at different states will 1) improve sample complexity; 2) increase the reachability of achievements that have not been unlocked. It is not obvious to me it will enable the agent to solve the fire realm, since the exploration bonus is still unstructured.
1
161
great research. what is the state in craftax? position, weapons, mana? and how much of this project was making it run fast vs understanding the algorithm, environment, why it failed?
1
2
317
The agent observation consists of a partial view of the env, full inventory and info about the agent itself (health, energy). As for my motivation, it was basically all of the above. I wanted to improve my systems knowledge and am interested in improving on-policy RL algorithms. Craftax is considered a difficult benchmark, so it seemed like a good exercise to work through the details and understand where PPO fails and why.
2
176
very cool!! love seeing how far tabula rasa RL can go :) your point about prior knowledge is also what excites me about LLM agents: we can communicate complex specifications (e.g how to fight pigmen) in natural language, they can learn in-context and even discover this knowledge autonomously via much more sample-efficient interaction with the environment, and they already bring priors about what's "high value" (useful) or interesting
1
5
412
"the really annoying sleeping mechanic" -- yeah, this is the biggest pain in Craftax imo @mitrma 😅
Astra solved it by REALLY emphasizing that it MUST always build shelter before sleeping. many failed attempts ended with the agent dying while sleeping
1
2
413
5 Exciting new results brought to you by our team + contributors: @spenccheng @finlay_sanders (fintern!) @ValtteriValo @daphnesolves @elliotarledge. Play with the agents at puffer.ai! All experimental data available is available in Constellation online + local.
3
5
77
6,220