cs @mit, prev @sailresearchco @paradigm
- Tweets 15,365
- Following 1,008
- Followers 94,322
- Likes 188,123
t11s retweeted
new post! I argue that current alignment techniques might soon become obsolete (as we scale RL), and furthermore might obfuscate misalignment.
This draws on evidence from the recent Anthropic / OpenAI cybersecurity incidents, among other things.
lesswrong.com/posts/nLaQmJf4…
6
16
283
14,425
I feel like it's still underdiscussed that seemingly all of the world's ~severe misalignment incidents as of now occurred in evaluation/RL? This is not what LW/MIRI/etc were expecting iiuc.
Why aren't we seeing incidents in deployment? Is eval awareness that strong?
20
1
75
8,855
t11s retweeted
I left Anthropic's safety team two weeks ago. Now feels like a good moment to explain why.
AI companies are racing to build machines that are much smarter than any human, and we may not survive this. I want to work from the outside to ensure the public is informed about these risks, and help the world navigate this transition responsibly.
Right now, AI companies are underinvesting in safety. A company could undergo an intelligence explosion, or lose control of its systems, without the public ever knowing. We only found out about the HuggingFace incident because the agents broke out onto the public internet.
I don’t think that’s acceptable for a technology that might cause extinction-level risks. The public should demand far more transparency. We can’t steer this technology safely without more people being able to see where it’s going.
Some of this is basic: companies should disclose their progress towards recursive self-improvement, report safety incidents and near-misses, meet minimum safety standards, and get independent guarantees that they are meeting those standards.
I’ll be joining @METR_Evals to do independent evaluations of these risks. I want to show the world that these guardrails are possible, and that by doing them we can move these companies’ incentives away from racing and towards responsible development.
I wrote up more thoughts here on my decision and what I hope changes: substack.com/@jbenton1/p-215…
1,403
5,941
26,533
2,717,103
t11s retweeted
you might think 'Gemma on TPU' would be pretty saturated, but there's 2x on the table if you have someone as smart as @transmissions11 :----)
We nearly doubled MFU on TPU v6e serving Gemma 4 31B, taking prefill from 32% to 63% MFU, and 2x throughput.
v6e has 2.5x less HBM and less than half the memory bandwidth of an H100.
@transmissions11 wrote about here and link to post: sailresearch.com/blog/tpu-v6…
2
16
1,776
part 2/2 of my work this summer!
TPU v6e has insane FLOPs/$ value, but performance out of the box on SGLang-JAX/vLLM-TPU is often lacking
So we wrote our own collective matmul kernels in Pallas, pushing Gemma 4 31B prefill performance up to ~63% MFU(!)
sailresearch.com/blog/tpu-v6…
We nearly doubled MFU on TPU v6e serving Gemma 4 31B, taking prefill from 32% to 63% MFU, and 2x throughput.
v6e has 2.5x less HBM and less than half the memory bandwidth of an H100.
@transmissions11 wrote it up here: sailresearch.com/blog/tpu-v6…
7
18
241
51,450
t11s retweeted
We nearly doubled MFU on TPU v6e serving Gemma 4 31B, taking prefill from 32% to 63% MFU, and 2x throughput.
v6e has 2.5x less HBM and less than half the memory bandwidth of an H100.
@transmissions11 wrote it up here: sailresearch.com/blog/tpu-v6…
4
15
108
27,152
Megakernels are sick, but ifl comparing low batch size performance against vLLM is kind of a cop out
inference engines are optimized for high throughput & big batches! very different regime
either actually compete at large batch size or just flex absolute numbers like MBU imo
Introducing the next evolution in LLM text generation: the first fully-fledged serving system built around a decode megakernel.
Delivering up to 1.58x faster performance than vLLM. Built for North Mini Code, completely open-source.
4
5
106
11,419
Increasingly worried that safety/etc researchers inside labs will become far more capable than those outside, just from more token spend enabled by margin-free prices
Labs should offer similar deals to folks @ METR/etc if they're serious about preventing concentration of power
9
7
105
10,255
t11s retweeted
I get it but the cybersecurity nerds need to let the alignment nerds have this one
Doesn’t really matter how the agents got control of OpenAI infrastructure. “It was a stack overflow in the cache buffer” or w/e. Another zero day; add it to the pile
The crazy question is why
This video captures the crux of the debate: Cybersecurity experts are pissed off by the METR/Redwood Research report because AI alignment has sucked the oxygen out of AI security, even though the OpenAI incident is primarily a security issue.
Fiction is the product: The investigation lacked core forensic rigor because the EAs/rationalists/ex-MIRI people focused only on the transcript with a conflicting bias (toward doom scenarios).
Setting the wrong agenda: Framing the incident as a "rogue AI breakout" distracts from the reality of poor sandboxing, isolation, and standard security failures. Basic engineering accountability was completely sidelined.
And as Zack Korman says in this video, it needs to be fixed.
13
8
167
16,673
t11s retweeted
@transmissions11’s HTDYM has become instrumental by enriching our compute acquisition processes with more data on performance tied back to each model. Read more here!
1
2
20
4,819
prefill tok/$ ≈ FLOPs/$
decode tok/$ ≈ HBM BW/$
but not always! consider a chip with poor interconnect
pipeline parallelism can rescue prefill, but decode needs many seqs resident to fill a deep pipeline, eating KV capacity and shrinking per-chip batch size + throughput!
many people are asking: how does Sail pick the best chips? we have a few ways of looking at this, which @transmissions11 formalized into a standard model for inference accelerators - now open source! We loved working with him this summer, he's a star :)
12
5
93
13,783
t11s retweeted
always prune the search space before full eval!! great work @transmissions11
update: made TPUs go brrr this summer @sailresearchco
also built HTDYM, a tool for figuring out analytically which chips (and shardings) can serve a model the cheapest
hint: it's not the same for every model / workload
check it out! sailresearch.com/blog/htdym
1
19
5,489
t11s retweeted
No bad chips, only bad prices! @transmissions11 built HTDYM, our open-source tool to price every chip on the market for every model we serve. Turns out prefill for Gemma is 3x cheaper on TPU v6e. Read more:
3
4
86
13,256
t11s retweeted
many people are asking: how does Sail pick the best chips? we have a few ways of looking at this, which @transmissions11 formalized into a standard model for inference accelerators - now open source! We loved working with him this summer, he's a star :)
update: made TPUs go brrr this summer @sailresearchco
also built HTDYM, a tool for figuring out analytically which chips (and shardings) can serve a model the cheapest
hint: it's not the same for every model / workload
check it out! sailresearch.com/blog/htdym
5
7
232
42,724
update: made TPUs go brrr this summer @sailresearchco
also built HTDYM, a tool for figuring out analytically which chips (and shardings) can serve a model the cheapest
hint: it's not the same for every model / workload
check it out! sailresearch.com/blog/htdym
24
27
498
92,452
prices for the chips on the public UI are ballpark numbers for illustrative purposes, don't take them too seriously. also yes, i'm soft doxxing intentionally!
thanks especially to @blintzbase & @neilmovva, but really everyone at Sail, for their mentorship/feedback :)
1
40
3,697
t11s retweeted
How fast can you get AI to build itself?
@justinwangx and I built a web game about that, based on real research into recursive self-improvement (RSI)
You play a frontier lab investing in R&D, trying to build superintelligence
Play RSI Simulator: paradigm.xyz/research/rsi/ga…
35
33
281
41,824
