Helping build AI dev tools at @weights_biases. I post about AI, data visualisation and the stuff I’m working on at wandb.
Dublin, Ireland
- Tweets 3,324
- Following 2,122
- Followers 5,859
- Likes 6,743
Pinned Tweet
Here's an animation of a @PyTorch DataLoader.
It turns your dataset into a shuffled, batched tensors iterator.
(This is my first animation using @manim_community, the community fork of @3blue1brown's manim)
Here's a little summary of the different parts for those curious:
1/5
34
487
2,591
Scott Condron retweeted
🪙 Platinum. Three reports in a row.
@SemiAnalysis_ published ClusterMAX™ 3.0 today, and CoreWeave is the only provider rated Platinum in all three editions.
The report: utm.io/us5Bv
8
12
121
8,675
Scott Condron retweeted
If you're curious why they only reported against TRL for their single GPU SFT on GLM Flash, it's because at that regime, they're still slower than Axolotl by about 5% (can't find any B300s, so using B200s). But hey, reporting benchmarks these days is just about cherry-picking data, right?
Introducing Halo, the best framework for post-training of open-source models.
Halo delivers up to 2.8x the throughput of stock TRL with less peak memory, while models stay in their native HuggingFace format.
Star us on GitHub: github.com/whitecircle/halo
3
2
36
4,553
This seems like a fun job
seeking those obsessed with evals
if you:
- are half researcher, half engineer
- love measuring hard to measure things
- are creative
- have a particular interest in post-training
...you'll like it here. sf based, in person, obsession coming from a place of curiosity. apply 👇
8
1,430
Scott Condron retweeted
We were thrilled to get early access to @typesafeai's Jev model and explore tons of new use cases!
Here's what @ktaletsk is building with Jev:
1. jevframe - pandas and polars dataframe adapter to bulk generate new columns with Jev github.com/ktaletsk/jevframe
Try the notebook: molab.marimo.io/github/ktale…
6
18
111
5,985
Scott Condron retweeted
Introducing Halo, the best framework for post-training of open-source models.
Halo delivers up to 2.8x the throughput of stock TRL with less peak memory, while models stay in their native HuggingFace format.
Star us on GitHub: github.com/whitecircle/halo
166
220
1,872
2,246,129
I invented Jev
Building a classifier in 2023
Use @OpenAI's new function calling API to define the possible outputs and then use the "input" argument it returns as the classification
5
30
3,136
Traces -> dataset -> distilled open model!
Join this webinar to learn about Model Distillation on top of your agent traces
Distillation, SFT, or RL?
The answer depends on your task, your eval bar, your timeline, and whether you have a research team. Usually not the model everyone defaults to.
40 minutes on how to decide, with a live walkthrough. Register: utm.io/usxVH
1
6
732
Scott Condron retweeted
The Training Agents series is completed!
It includes 6 different videos:
> Agentic Evaluations Workshop
> RL for Agents Workshop
> Training Agents 1: SFT on agent traces
> Training Agents 2: Distillation
> Training Agents 3: Reinforcement learning
> Training Agents 4: RL environments
8hrs of live material with some videos over 200k views, go watch it!
piped.video/playlist?list=PL…
6
83
567
29,744
a new challenger approaches
Glad to see another live training run! Here's how Marin 535B-A23B is doing today: wandb.ai/marin-community/mar…
Anyone else want to share?
1
2
615
Scott Condron retweeted
Glad to see another live training run! Here's how Marin 535B-A23B is doing today: wandb.ai/marin-community/mar…
Anyone else want to share?
Nearly half a year of silence. We spent it studying one problem: how far RL can scale.
MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks.
Streaming the run: mimo.xiaomi.com/rl/
32
71
843
76,447
Scott Condron retweeted
Building a classifier in 2023
Use @OpenAI's new function calling API to define the possible outputs and then use the "input" argument it returns as the classification
32
149
1,139
547,171
Scott Condron retweeted
at openai, I used to say that if you don't provide a logit_bias for the functions instead of the tokens, you don't give a f*ck about automation
3
1
42
2,864
How far we’ve come
Building a classifier in 2023
Use @OpenAI's new function calling API to define the possible outputs and then use the "input" argument it returns as the classification
2
1
58
7,760
I think this is also why auto-research agents haven't really stuck - especially the part about memory & harnesses needing to be tightly integrated.
memory has been a hot topic for the past ~2 years
every time we bring it up or do anything there, gets ton of interest, but nothing sticks
few takes on why below:
- how to update and then use memory needs to be pretty tightly integrated with the harness. so its hard to build a standalone memory product
- the hardest part of memory is the logic of deciding what to remember. storing and even querying memory is not necessarily hard. the logic of deciding what to remember is often (a) application specific, (b) just prompting, which makes it hard to productize
- memory is not that useful in a general purpose agent, but more for when you are doing repeated tasks. more high usage ai agent are not just doing the same task repeatedly
- memory hasnt really proven to be useful for coding agents, and most agent things that take off happen first in coding agents
2
4
602
Scott Condron retweeted
Distillation isn't a research project anymore. It's a pipeline.
Weave traces → dataset curation → Serverless SFT → eval → deployment.
Join us for 40 minutes on turning real production usage into a smaller student model that meets your bar. utm.io/usxVH
1
3
38
3,066
I think one of the reasons why teams struggle to get AI apps/agents into prod is because it’s more like research than sprint-planned product dev.
Most time in AI-product / model development isn’t spent at the harness layer or tuning prompts on your clearly defined task - it’s iteratively building measurable tasks so you gradually get closer to your production use-case / goal
This quote from
@johnschulman2 on @dwarkesh_sp podcast describes this well:
> A lot of research isn’t hill-climbing on a well-defined goal. It’s an intuition on some way models could be better and we have some idea for an algorithm that would go in that direction. Let’s come up a task that could show signs of life on this approach. Then make successively more realistic versions of the task.
> You’re not directly optimising for the production objective, you’re relaxing the realism-axis for a bit, and find some methods that actually work and get back to realism later after the method matures a bit. (at 30 mins)
New episode with @johnschulman2, @oneill_c and @BerenMillidge.
I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next.
0:00:00 – Steelmanning the case against RSI
0:18:39 – What’s driving the Chinese labs’ progress
0:28:06 – How will automated AI researchers be trained
0:33:51 – Will long-horizon RL elicit AGI?
0:45:24 – The sim-to-real gap
1:00:33 – How much progress is explained by data?
1:18:03 – Why is RL working so well?
1:24:54 – Move 37 and entropy collapse
1:28:31 – Rapid-fire timelines
2
1
1
764
Scott Condron retweeted
Next up Julia showcasing @CoreWeave @wandb ARIA!
Winner for best use of ARIA gets $1,000 as well!
Check out ARIA: wandb.ai/site/agent/
3
6
898
Scott Condron retweeted
We are underway @CoreWeave Hackathon!
@neutralino1 walking through @wandb Weave! With a prize of $1,000 for best Weave use this weekend.
For those participating- Many more prizes like a robo dog and f1 tickets to come!
3
5
173