my latest blog post "auto-research with codex: how I achieved a 212x faster kernel over baseline with codex in GPU Mode's qr_v2 problem" is up now. in this post, i talk about my approach towards auto-kerneling on the QR decomposition problem. sankalp.bearblog.dev/autores…

We're open-sourcing Mixture-of-Kittens (MoK), our MoE training megakernel for NVL72s. It fuses all Mixture-of-Experts communication and computation into a single, fully deterministic kernel, and runs up to 2.37x faster than the strongest public baselines.

I stand on the shoulders of giants who stood on the shoulders of giants. So much of the progress we’ve made has been because of the openness of our great predecessors. Proud to support microsoft.com/en-us/corporat…

Played around with this a bit. It’s nice to see folks thinking about it, and it is certainly an important problem to address. It's a deceptively good benchmark that asks whether gradient-based learning can learn a procedure for reusable computation that generalises to more steps than it saw during training, or whether it just learns a map between inputs and outputs. The question is whether the model can learn the serial computation itself, rather than relying primarily on test-time compute (in token space), and whether latent depth can provide a cheaper form of sequential computation. You can see the thought that has gone into designing the tasks. Kudos to folks @coreauto and @tilderesearch.

Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable? @coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper" We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases. This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way. The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation. Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!

I'll be leading a sprint on the @GPU_MODE Cholesky competition at @europython / @EuroSciPy this Sat/Sun in Krakow! Come learn how to autoresearch and write CUDA kernels. @modal has generously donated GPU access and @nvidia will provide frontier LLM access for the sprint.

Launching a new kernel competition: Linear Algebra Kernels For The Age Of Research. First problem: batched QR decomposition on B200. Old math, modern hardware. Prize: Rare swag and hangout in SF

We've released the QR problem, a more robust qr_v2 with a fresh leaderboard so please resubmit! Thank you to @blelbach, @myainotez and @nikhilbarhate99 for sharing feedback. Sorry if I missed anyone! I considered automatically backfilling all submissions but the rankings do change quite a bit so I figured a refresh would be better. Changelog * Fail submissions if they fail when we change random seeds * Add nasty correctness cases with more degenerate inputs in mixed batches * Recheck correctness when doing perf testing to avoid Volkswagen cheat * Reject Nan/Inf residuals * Validate each matrix factorization residual, since averaging was hiding bad matrices * Old QR is still open so folks can't see submissions but you can't submit anything to it Wontfix * Stream hacking is still banned via very blunt ban of the word "stream" we don't have a good solution for this * CUDA graphs are allowed but not particularly interesting to us Best submissions so far if I resubmit their solutions are

We've released the QR problem, a more robust qr_v2 with a fresh leaderboard so please resubmit! Thank you to @blelbach, @myainotez and @nikhilbarhate99 for sharing feedback. Sorry if I missed anyone! I considered automatically backfilling all submissions but the rankings do change quite a bit so I figured a refresh would be better. Changelog * Fail submissions if they fail when we change random seeds * Add nasty correctness cases with more degenerate inputs in mixed batches * Recheck correctness when doing perf testing to avoid Volkswagen cheat * Reject Nan/Inf residuals * Validate each matrix factorization residual, since averaging was hiding bad matrices * Old QR is still open so folks can't see submissions but you can't submit anything to it Wontfix * Stream hacking is still banned via very blunt ban of the word "stream" we don't have a good solution for this * CUDA graphs are allowed but not particularly interesting to us Best submissions so far if I resubmit their solutions are

I have some mixed feelings about this result: On the one hand, it's genuinely impressive. I didn't know that Shampoo could be configured to perform this well on the benchmark. On the other hand, the way this performance boost was achieved seems difficult to call "Vanilla," for the following reason: According to @_arohan_, the boost depends upon fixing a numerical linear algebra issue that he observed to occur in my initial standard DistributedShampoo run. He fixed the issue by enabling the flag rank_deficient_stability_config=PseudoInverseConfig(). Here's the problem: This is an undocumented flag. It is contained within the 12,000-line DistributedShampoo codebase, but it does not appear in any user-facing documentation. As a result, if someone tries to train a model using DistributedShampoo without either (a) knowing about this special undocumented flag or (b) being prepared to detect and fix the numerical linear algebra issues that may occur without it, then they won't be able to achieve @_arohan_'s level of Shampoo performance. This level of effort would be considered atypical for mere hyperparameter tuning. -- [Note on Muon baseline in plot below: Rohan's post compared Shampoo to a slightly undertuned Muon baseline from 2026/05/01, which reached the target loss in 3375 steps. This resulted in a 50-step gap between Shampoo and Muon. In the figure below I'm using the up-to-date 2026/05/03 baseline, which reaches the target in 3325 steps. This results in the step-counts exactly matching between Muon and the tuned/stabilized Shampoo variant.]

June 9th Researcher Reciprocity License "if you train on it, you let us generate - reverse terms of use void" Status quo 1. We teach frontier devs with ICLR/NeurIPS papers, OSS Github contributions 2. They use it to make frontier models 3. Then ban us from exploring our ideas We need a new license, original thinkers can't be an underclass to a tyrannical researcher fiefdom