grand theft alignment

MiMo-V3 is getting a new architecture. The core of it, HySparse2, is out today. Less prefill, a smaller KV cache, better long-context retrieval—and we got all three at once. Compared with MiMo-V2.6's Hybrid SWA architecture: • 5.02× lower prefill FLOPs at 1M tokens • 4.5× smaller KV cache at 1M tokens • Better MRCRv2 and RULER-v2 scores, plus lower AgentPPL and LongPPL Why build a new architecture? Agentic inference is a very different workload. Each round, a short action can return a long observation that needs to be prefilled, while the context keeps growing. That puts prefill cost, KV-cache size, and retrieval accuracy on the critical path at the same time. HySparse2 tackles all three with two levels of KV sharing: • KV Bridging: Following YOCO, full-attention layers in the cross-decoder build their K/V from self-decoder hidden states. • KV Reuse: Within each hybrid block, sparse layers reuse the preceding full-attention layer's KV cache and selection indices. Two more changes: token-level selection replaces block-level selection, and a forced window of recent tokens replaces the separate SWA branch, so local and global tokens share one KV cache. Since all cross-decoder KV caches now come from the self-decoder, prefill can stop once the self-decoder finishes. Paper: arxiv.org/pdf/2609.26368

the singularity is truly amazing

gpt-6 somehow has me more hyped than gta-6

The only official way to see the 1.43 IMAX 70MM sequences in THE DARK KNIGHT at home are on the special edition HD Blu-ray set. This is also a good demonstration of how drastic the change is from 2.39 to 1.43 on a home setup.

My book, Reinforcement Learning from Human Feedback is done! This is the book I wish I had when learning to fine-tune, align, & now post-train models since ChatGPT. The resource has been built by me finding time to study and document the fundamentals on nights and weekends since 2024. Transferring as much of the intuitions of building Olmo as I possibly can in the book format. The book is launching with an over 10 hour, full course with slidedecks, functional code for the training chapters, an example model completions library, and of course the free online web version. Physical orders from Manning will ship in 1-2 weeks, and Amazon a week or so after. Thanks for your support!

drive-by humiliation of all mathematicians he disproved a 87 year old open math conjecture by prompting Fable and the counterexample is almost trivial

Here's XEphem the interactive astronomy program for UNIX platforms

It's fucking over. Anthropic got J Cole Pack it up Sam. Good game.

8 months, Q1 2027 the bigger problem is that Mythos itself is a last generation pretrain, and by then Anthropic will likely have completed something vastly more capable, so the real gap will only increase, towards Dario's targeted 24 months.

how it feels to be a hardware engineer visiting shenzhen

Hear me out, vim, but you use this to switch modes:

$650K for your own personal mecha that can go Gorilla Mode Imagine the customization… No idea who's buying this but its pure childish awesomeness can't be denied

wow

this is art

MAY RAM NEVER BE CHEAP AGAIN

Companies with semi-frontier or frontier labs, from memory: Alibaba (2+ labs), Ant, BOSS Zhipin, ByteDance, DeepSeek, Meituan, MiniMax, Moonshot, OpenBMB, RedNote, StepFun, Tencent, Ubiquant, Xiaomi, zAI Any of these could rival Anthropic if given 300K H200s and 6 months

man, it turns out all you really need to bootstrap a frontier lab is impeccable research taste, world class engineering talent, a single pretrained base model and enough compute to both serve and train that model at the same time

that is not a swarm. this is a swarm.