Building SkyRL at @BerkeleySky Join the Slack community: join.slack.com/t/skyrl/share…
Berkeley, California
- Tweets 160
- Following 18
- Followers 2,839
- Likes 250
It was a packed house at the SkyRL Meetup today.
Thanks to all the speakers!
Join us for the first-ever SkyRL Meetup!
It's been a year since we released the SkyRL library. SkyRL now has 2k+ GitHub stars and a growing user base in academia and industry, including researchers from Stanford, CMU, UC Berkeley, Microsoft AI, Datadog, and more.
We're hosting a SkyRL Meetup on August 18th at the Anyscale office in SF. We have a special set of lightning talks:
🔗 Making frontier RL training accessible to everyone, from @trajectorylabs
🐶 Post-training an SRE agent with SkyRL, from @datadoghq
⚡️ Scaling RL with SkyRL on AMD GPUs, from @AMD
🧑🏻💻 Training knowledge work agents with SkyRL, from the SkyRL team (@charlie_ruan)
We’ll also do a deep dive into the library and talk about the road ahead. More details here: luma.com/y4ahouh6
2
12
4,941
To get started with `skyrl-train` as a library instead of forking, you can follow our instructions here: skyrl.readthedocs.io/en/late… to use the PyPI package!
(6/n)
1
2
316
Recipes: We've compiled a list of end to end recipes for reproducible runs on math (DAPO), Text2SQL, and search tasks!
Recipes page: skyrl.readthedocs.io/en/late…
(3/n)
1
127
AsyncRL: We support fully async training (aka "in-flight weight updates" or "multi-turn partial rollout") for any Generator that subclasses our GeneratorInterface for creating trajectories!
Docs: skyrl.readthedocs.io/en/late…
(2/n)
1
1
152
We recently released SkyRL-Train v0.3.0!
Highlights include:
- Experimental support for Pipeline-RL style Async-RL
- Updated E2E Recipes page with Math, Search, SQL runs
- Migration from mbridge -> Megatron-Bridge
- 14 new OSS contributors!
(1/n) 🧵
2
6
28
3,190
☁️SkyRL now runs seamlessly with SkyPilot!
Let @skypilot_org handle GPU provisioning and cluster setup, so you can focus on RL training with SkyRL.
🎯 Launch distributed RL jobs effortlessly
⚙️ Auto-provision GPUs across clouds
🤖 Train your LLM agents at scale
Get started with the SkyRL × SkyPilot guide:
🔗skyrl.readthedocs.io/en/late…
🔗docs.skypilot.co/en/latest/e…
9
25
5,497
SkyRL v0.2.0 is here!
With 22 contributors (including 11 new contributors!), this release holds many updates like strong MoE support with Megatron, LoRA support, standardized inference on the OpenAI API, new integrations, and many many more.
The code: github.com/NovaSky-AI/SkyRL
1
4
23
2,187
SkyRL just crossed 1000 Github stars! Thank you to all the wonderful contributors and users building this project together 🥳
Check it out: github.com/NovaSky-AI/SkyRL
5
37
5,967
Scaling agentic simulations is hard, so in collaboration with @anyscalecompute we wrote up our experience using Ray for agent sims—featuring an end-to-end RL example with SkyRL!
Check it out here: anyscale.com/blog/massively-…
1
5
11
3,079
3/N
We train three models with different maximum tool-call turns (2, 3, and 4), but a fixed total token generation budget. We observe that the model learns to issue more search queries for more complex questions, showcasing the benefits of multi-turn search.
1
4
415
2/N
In the guide, we provide a step-by-step walkthrough of implementing a multi-turn search environment in the simple SkyRL-Gym interface, with no modifications needed in the training stack.
1
3
597
🔎 SkyRL + Search-R1
Training a multi-turn search agent doesn’t have to be complicated.
With SkyRL, reproducing the SearchR1 recipe at high training throughput is quick and easy! We wrote up a detailed guide to show you how:
novasky-ai.notion.site/skyrl…
1/N 🧵
2
32
145
17,954
(6/N) SkyRL-v0.1 introduces SkyRL-Gym – a lightweight gymnasium of tool-use environments with a simple interface and a library of built-in environments for math, coding, search, and text-to-SQL.
1
1
11
501
(4/N) As evidence for SkyRL’s extensibility, we provide several examples modifying the stack, like:
Implement a new environment in <50 LoC.
Update the sync training loop into async one-off pipelining in <100 new LoC.
Disaggregate training and generation on heterogeneous HW with only a config change!
1
9
636
✨Release: We upgraded SkyRL into a highly-modular, performant RL framework for training LLMs. We prioritized modularity—easily prototype new algorithms, environments, and training logic with minimal overhead.
🧵👇
Blog: novasky-ai.notion.site/skyrl…
Code: github.com/NovaSky-AI/SkyRL
2
47
203
43,896
6/N 💡 We observe that the model learns to
📖 Break problems into sub-steps
🔍 Verify intermediate results
🛠 Fix syntax + logic errors in SQL
♻️ Iterate on failures
Real examples show the model debugging its way to correct SQL.
1
13
365
5/N 💡 Multi-Turn vs. Single-Turn RL
⚡ 2.8× fewer training steps to the same reward
🎯 +16% higher reward after 35 steps
Even in 1-turn eval w/o feedback, it performs better: showing multi-turn improves not just the ability to leverage feedback but also overall reasoning capability.
1
11
756
4/N We train SkyRL-SQL-7B on top of Qwen2.5-Coder-7B-Instruct with:
• ✅ 653 samples
• ✅ Simple rewards (format + execution)
• 🚫 No partial rewards or million-scale data
Trained for 5 turns and 14 epochs.
Result? 📈
+7.2% gain over base
+1.6% over GPT-4o
+1.8% over o4-mini
1
11
496
3/N We built a multi-turn RL pipeline on top of VeRL and SearchR1 for Text-to-SQL — letting LLMs think, query the database, observe results, refine, and output final solutions.
1
12
553
2/N Generating SQL in one shot often fails when questions or schemas get complex. Inspired by exploratory data analysis, we train LLM agents to iteratively refine SQL through trial and error — not rely on one-shot guesses.
1
12
857
1/N Introducing SkyRL-SQL, a simple, data-efficient RL pipeline for Text-to-SQL that trains LLMs to interactively probe, refine, and verify SQL queries with a real database.
🚀 Early Result: trained on just ~600 samples, SkyRL-SQL-7B outperforms GPT-4o, o4-mini, and SFT model trained with 2.5M samples across multiple Text-to-SQL benchmarks!
🧵👇
Code: github.com/NovaSky-AI/SkyRL
Blog: novasky-ai.notion.site/skyrl…
Model and Data: huggingface.co/collections/N…
5
32
146
33,323
5/N To mitigate the slow generation problem, we introduce an asynchronous rollout with a 3-stage producer-consumer pipeline. This brings 4-5x speedup compared to a naive synchronous implementation.
1
1
9
917
4/N To enable scalable environment interactions, we implement a scalable remote sandbox server. We deploy our server on Kubernetes, leveraging storage-optimized instances to cache container images and enable fast startup times.
1
1
11
1,512
1/N Introducing SkyRL-v0, our RL training pipeline enabling efficient RL training for long-horizon, real-environment tasks like SWE-Bench. We also open-source a series of our early trained models to showcase the potential of end-to-end online RL training on long-horizon (20-50 turns) agentic tasks.
- SkyRL-Agent-7B-v0: 11.0% → 14.6%
- SkyRL-Agent-8B-v0: 3.6% → 9.4%
- SkyRL-Agent-14B-v0: 18.0% → 21.6%
Code: github.com/NovaSky-AI/SkyRL
Blog: novasky-ai.notion.site/skyrl…
Models: huggingface.co/collections/N….
2
70
264
95,861
5/8 🥰
Integrating both ideas, S* outperform majority-voting based or previous self-debugging based approaches.
1
4
602
3/8 🛠️
S* is simple and extensible:
(1) Parallel + Sequential Scaling with iterative debugging from public test execution feedback.
(2) Execution grounded selection: a hybrid interpreter + LLM selection mechanism reliably and efficiently select the best solution.
1
2
7
786
1/8 🚀
Introducing S*: Test-Time Scaling for Code Generation, start of our releases in the coding domain @NovaSkyAI .
S* enables (1) non-reasoning models surpass reasoning models: GPT-4o-mini + S* > o1-preview. (2) open models compete SOTA: R1-Distilled-32B +S* ~= o1 (high).
6
47
200
25,023
📝 Sky-T1-mini training recipes:
- Base: DeepSeek-R1-Distill-Qwen-7B
- Data: 30K subset from STILL3 and Eurus-2-RL-Data
- Train: Simple RLOO with 4 rollouts per prompt
- Cost: 36 hours on 8xH100, around $870 according to
@LambdaAPI
1
2
16
805
We observe that the Long CoT SFT in general enhances the model’s pass@k performance while RL lifts the model’s performance at lower generation budgets (i.e., pass@1), which sometimes comes at a cost of the entropy of solutions. For more pass@k analysis, refer to our blog!
1
1
11
885
🛠️ Sky-T1-7B training recipes
- Base: Qwen2.5-Math-7B
- Step 1: SFT on 5K math reasoning traces from QwQ.
- Step 2: RL on 30K prompts.
- Step 3: SFT on 5K samples distilled from Step 2 checkpoint + 5K math reasoning traces from QwQ.
- Step 4: RL again on 15K harder prompts.
1
1
16
1,068
Interleaving SFT + RL unleashes the power of small models.
🚀 Introducing Sky-T1-7B, a SOTA open-recipe model trained with 4-step SFT->RL->SFT->RL. We also release Sky-T1-mini, trained with RL from DeepSeek-R1-Distill-Qwen-7B model, approaching o1-mini performance on math tasks.
6
52
279
30,902
4/5 ⚙️
The training recipe:
- Base: Sky-T1-32B-Preview
- Data: Curated from Sky-T1-32B-Preview, filtered and rewritten responses to create {positive: short len, negative: long len} preference pairs
- Train: Simple Policy Optimization (SimPO)
- Cost: 8 H100 GPUs, 11.5 hours, $275
2
9
777
3/5📈
Sky-T1-32B-Flash reduces generation lengths and maintains accuracy across domains. Relative to Sky-T1-32B-Preview:
- Math500: 33% fewer tokens
- LiveCodeBench-Hard: 57% fewer tokens
- GPQADiamond: 39% fewer tokens
1
8
303
1/5 ⚡️Presenting Sky-T1-32B-Flash⚡️, our open reasoning model that tackles "overthinking" to cut generation lengths (and inference cost!) by 50% without sacrificing accuracy – tuned with only $275!
📊Blog: novasky-ai.github.io/posts/r…
🏋️♀️Weights: huggingface.co/NovaSky-AI/Sk…
5
18
63
12,794
4/6 ⚙️
The training recipe:
- Base: Qwen2.5-32B-Instruct
- Data: Curated from QwQ-32B, enhanced with GPT-4o-mini, reject sampling for high-quality math & coding reasoning traces.
- Cost: 8 H100 GPUs, 19 hours, $450.
16
10
103
13,543
1/6 🚀
Introducing Sky-T1-32B-Preview, our fully open-source reasoning model that matches o1-preview on popular reasoning and coding benchmarks — trained under $450!
📊Blog: novasky-ai.github.io/posts/s…
🏋️♀️Model weights: huggingface.co/NovaSky-AI/Sk…
123
233
1,230
312,780