Joseph Suarez 馃悺
@jsuarez

I build sane open-source RL tools. MIT PhD, creator of Neural MMO and founder of PufferAI. DM for business: non-LLM sim engineering, RL R&D, infra & support.

puffer.ai
Joined March 2019
793 Photos and videos
Releasing PufferLib 5.0: Train agents in under a second
47
154
1,827
165,537
Joseph Suarez 馃悺 retweeted
Finlay Sanders
@finlay_sanders
Sep 14
Article

Beating AutoAscend鈥檚 NetHack Challenge Score with RL

NetHack is a game of long horizons, partial observability, and brutal interactions that can end a run with little warning. This is rough if you're learning to play, but makes for an unbelievably

4
15
109
25,048
If you're using raylib professionally, get your company to pitch in a couple hundred bucks a month. Or do it yourself.
Ray
@raysan5
Sep 22
How is it possible to get that level of open source support while many other projects like raylib struggle to get enough finance to even pay a developer minimal salary? I don't know... 馃槥
1
83
7,163
Article

You have SKILL_ISSUES.md

Contributors send PRs. I spend time, as Elon put it, "fixing their fucking stupid code." I semi-automated it in a "Skill" file, the naming of which is the most brilliant piece of pure-marketing

6
11
279
19,267
Article

PufferLib 5.0's Streamlined RL Algorithm

TL;DR: Removed VTrace, prioritized advantage filtering, and advantage normalization. The destination was simple, but the journey contradicts a major line of work that I had not even thought to

3
15
138
8,887
Two tasks, ~30 minute and ~90 minute horizon respectively. Typically completed by the top 1-3% of players after 10's of hours of attempts. These results changed my understanding of what PufferLib is capable of doing!
valo
@ValtteriValo
Sep 14
i'd like to keep it low key and say this is no big deal, but in reality this has to be the coolest project i've worked on maybe ever, and there's a lot more to come!
Article

Old School RuneScape in PufferLib 5.0

banner image: final boss enraged, with ~unavoidable damage pummeling the player, while trying to stay behind a moving shield In Old School RuneScape a game tick has happened every 600 milliseconds

1
4
116
8,058
Note: we do NOT run any of this on the live game servers! It's all sim. OSRS is my favorite game and I hate seeing the econ + major account achievements ruined by botting
16
729
>Double RL SOTA w/ a very vanilla setup. Probably the first major point of comparison to existing lit for PufferLib! I consider this saturated without tweaking the enchant system. The current impl doesn't really fit an RL benchmark.
Daphne Cornelisse
@daphnesolves
Sep 14
Article

Lessons from reaching level 6 in Craftax through tabula rasa on-policy reinforcement learning

The policy below is playing Craftax. It obtains a total episode return of 162, which is 71% of the maximum achievable return. It crafts armour, learns to cast fireball, and defeats numerous enemies
2
53
5,711
Spencer took point on stable self-play this update. Robocode is an awesome RL benchmark. Super fast to sim, lots of interesting strategies, plenty of hard scripted opponents to beat!
Spencer Cheng
@spenccheng
Sep 14
Article

Selfplay in Games Without a Built In Curriculum

Two tanks enter a 2D arena. One lives. Can a policy trained only on copies of itself beat experts it has never seen. In PufferLib 5.0, it can. We test this in Robocode, a 2000s coding game where
3
48
4,641
Joseph Suarez 馃悺 retweeted
valo
@ValtteriValo
Sep 14
i'd like to keep it low key and say this is no big deal, but in reality this has to be the coolest project i've worked on maybe ever, and there's a lot more to come!
Article

Old School RuneScape in PufferLib 5.0

banner image: final boss enraged, with ~unavoidable damage pummeling the player, while trying to stay behind a moving shield In Old School RuneScape a game tick has happened every 600 milliseconds

9
10
81
14,453
Article

Solving Breakout in Under a Second with PufferLib 5.0

This is the in-depth engineering article accompanying PufferLib's 5.0 release: up to 60M step/second useful single-GPU training. In 4.0, we broke the mold with a custom CUDA C stack over a grueling

4
6
107
4,798
>5x RL SOTA on NetHack with PufferLib 5.0 and still SOTA over the best hand-coded scripted agent. Awesome work by our Fintern this summer!
Finlay Sanders
@finlay_sanders
Sep 14
Article

Beating AutoAscend鈥檚 NetHack Challenge Score with RL

NetHack is a game of long horizons, partial observability, and brutal interactions that can end a run with little warning. This is rough if you're learning to play, but makes for an unbelievably
4
88
6,263
Double SOTA perf for RL on Craftax with PufferLib 5.0
Daphne Cornelisse
@daphnesolves
Sep 14
Article

Lessons from reaching level 6 in Craftax through tabula rasa on-policy reinforcement learning

The policy below is playing Craftax. It obtains a total episode return of 162, which is 71% of the maximum achievable return. It crafts armour, learns to cast fireball, and defeats numerous enemies

2
7
104
8,164
Article

Reinforcement Learning with 5 Degrees of Parallelism

PufferLib 5.0 broke 60M step/second training on a single RTX 5090. For most environments, that is months of simulated data every second. This is how we did it. Parallelizing Environments High-perf

5
28
283
14,268
Article

The 5 Coolest RL Results in PufferLib 5.0

5. Multitask Drone Policies One single policy is able to both race and arrange into several different formations. Brought to you by our intern, Fin. Or should I say fintern... very appropriate for

6
25
247
25,492
Releasing PufferLib 5.0: Train agents in under a second
47
154
1,827
165,537
more replies
We now have ~60 environments in Ocean available directly on puffer.ai. Almost all have strong baselines that run right in your browser. Locally, not on our servers. Because you can do that with models this small!
1
1
66
3,565
Everything you see here is free and open source. We also build ultra high performance environments and agents for non-LLM use cases professionally and have applied our tech in gaming, animation, finance, commerce, and defense. Contact us to puff up your training!
2
2
76
3,120
Load more