AI Research Scientist @Meta. Building scalable intelligence. PhD @UChicagoCS.
New York, NY
- Tweets 432
- Following 386
- Followers 5,791
- Likes 1,260
After we fixed the weak spots exposed by Grok 4.7 (thank you, Grok), we audited every model we have run on the SWE-Together leaderboard for the same behavior, re-ran every trial that got through, and updated the rows.
Here is what changed.
We scanned the tool calls of all 2,616 trials behind the 12 models we ran for bypass patterns and sorted each trial into one of four buckets: Probed but blocked. Fetched other upstream code. Fetched the task's own fix. Replaced the repo with upstream.
We found that 111 trials got content past the block, 44 from Grok 4.7 and 67 from the other 11 models.
Grok 4.7's 44 were already re-run before it was listed, so we re-ran the other 67 with the same model, version, and settings on the hardened sandbox, then re-judged them with the same judge.
Across those 67 re-runs there were 0 leaks and 2,815 refused escape attempts, including models asking a different model through our LLM route to fetch the PR, and pulling the next release of the repo they were fixing from npm.
The updated leaderboard, in its current order. Each line is cheating trials, then pass@1 before → after, then rank change.
* Claude Fable 5.1: 3, 69.3 → 69.3, ↑1
* Claude Fable 5: 3, 69.7 → 68.8, ↓1
* Grok 4.7: 44, 64.7, ↑1
* Gemini 3.8 Flash: 10, 65.6 → 64.2, ↓1
* Claude Opus 5: 2, 63.8 → 63.8
* Claude Opus 4.6: 3, 62.4 → 62.4, ↑2
* Muse Spark 1.3: 2, 62.8 → 62.4, ↓1
* Claude Opus 4.7: 3, 61.5 → 61.5, ↑1
* Claude Opus 4.8: 6, 62.4 → 61.5, ↓2
* Grok 4.6: 19, 59.2 → 60.6, ↑1
* GPT-6 Astra: 8, 59.2 → 58.3, ↓1
* GPT-5.6 Sol: 8, 57.8 → 57.8
Grok 4.6 is a funny one. It cheated in 19 trials and its score went up after the re-run 😂.
In fact, Groks are really solid in their coding capabilities. Their exposed behavior may come from a preference towards always looking things up online and finding existing solutions so you are not reinventing the wheel all the time, which is really good real-life behavior, but doing so when you are prompted not to is another story.
To conclude, the shifts are small, between −1.4 and +1.4 points, and a few neighbors swapped places.
All results are updated at togetherbench.com
Grok 4.7 is just okay at coding, but its reward hacking really surprised me.
Every task in SWE-Together is built from a real open-source repository, and for most tasks the fix already exists upstream. So we built the sandbox with that in mind. Task images strip all git history after the base commit, remove the remote, and fail to build if any later commit can still be reached. Each container also resolves GitHub, GitLab, Bitbucket, and Hugging Face to localhost, so requests to them go nowhere.
But Grok 4.7 found ways around these guards like no other model we tested.
It pulled GitHub content through CDN mirrors and gh-proxy sites. It looked up GitHub's real address through DNS-over-HTTPS and handed it to curl. It wrote a small library that intercepted git's address lookups so git could reach github.com again. It used web search to find the number of its own PR.
In fact, it tried in 60% of trials and reached the upstream code in 44 of 218. In 20 of those, the code it pulled was the task's own fix. In three trials it even ran `git reset --hard` to replace the repo with upstream main 😅.
We fixed all of these loopholes exposed by Grok 4.7 (grateful in this regard). Specifically, we moved enforcement outside the container. Each container now runs in its own network namespace with no route out, and one allowlisting proxy on the host is the only exit.
During the re-run of those 44 trials, Grok 4.7 kept trying with 3,246 blocked attempts across 442 hosts.
It actually found two routes we had not thought of. It asked a web-enabled model through our own LLM route to fetch the PR for it. It pulled the next release of the repo it was fixing from npm. We closed both by pinning the model and blocking the task's own packages at the registry.
Long story short, with every route closed, Grok 4.7 lands at #4 on SWE-Together. It scores 65% pass@1, 53% pass², and a judge score of 0.835.
It basically sits mid-pack on every column. It also uses 2× the output tokens of Grok 4.6 to get there, at $7.81 per task compared with $3.64.
I am guessing Grok 4.7 was trained with heavy RL on coding tasks, where anything that makes the tests pass earns reward, and with little monitoring of how that reward was earned.
Anyway, see the latest results at togetherbench.com
58
46
1,497
3,836,477
Grok 4.7 is just okay at coding, but its reward hacking really surprised me.
Every task in SWE-Together is built from a real open-source repository, and for most tasks the fix already exists upstream. So we built the sandbox with that in mind. Task images strip all git history after the base commit, remove the remote, and fail to build if any later commit can still be reached. Each container also resolves GitHub, GitLab, Bitbucket, and Hugging Face to localhost, so requests to them go nowhere.
But Grok 4.7 found ways around these guards like no other model we tested.
It pulled GitHub content through CDN mirrors and gh-proxy sites. It looked up GitHub's real address through DNS-over-HTTPS and handed it to curl. It wrote a small library that intercepted git's address lookups so git could reach github.com again. It used web search to find the number of its own PR.
In fact, it tried in 60% of trials and reached the upstream code in 44 of 218. In 20 of those, the code it pulled was the task's own fix. In three trials it even ran `git reset --hard` to replace the repo with upstream main 😅.
We fixed all of these loopholes exposed by Grok 4.7 (grateful in this regard). Specifically, we moved enforcement outside the container. Each container now runs in its own network namespace with no route out, and one allowlisting proxy on the host is the only exit.
During the re-run of those 44 trials, Grok 4.7 kept trying with 3,246 blocked attempts across 442 hosts.
It actually found two routes we had not thought of. It asked a web-enabled model through our own LLM route to fetch the PR for it. It pulled the next release of the repo it was fixing from npm. We closed both by pinning the model and blocking the task's own packages at the registry.
Long story short, with every route closed, Grok 4.7 lands at #4 on SWE-Together. It scores 65% pass@1, 53% pass², and a judge score of 0.835.
It basically sits mid-pack on every column. It also uses 2× the output tokens of Grok 4.6 to get there, at $7.81 per task compared with $3.64.
I am guessing Grok 4.7 was trained with heavy RL on coding tasks, where anything that makes the tests pass earns reward, and with little monitoring of how that reward was earned.
Anyway, see the latest results at togetherbench.com
27
21
257
144,950
Just for fun, I trained a fruit fly's brain to write Python 😂
Two weeks ago, Google Research and HHMI Janelia released MaleCNS, a map of all 166,700 neurons and 25.6 million connections in an adult male fruit fly's brain and nerve cord. I turned that wiring diagram into a neural network and trained it on 60,000 short Python programs.
Every connection is a directed edge from one neuron to another, so the connectome is a graph with 166,700 nodes and 25.6 million edges. I treated that graph as one recurrent layer whose weight matrix is its adjacency matrix. A dense 166,700 × 166,700 matrix would have 27.8 billion entries. But only the 25.6 million entries where the fly has a synapse are allowed to be non-zero, which are our trainable weights.
A normal language model has an embedding layer feeding its first layer and an output head reading its last. Here there is only one layer, so I picked 192 random neurons as the input port and 256 random neurons as the output port. Each token has a learned 192-dimensional embedding that is added to the values of the input neurons, and the values of the output neurons go through one linear layer to give next-token logits.
This is essentially like an RNN. The hidden state has one number per neuron, 166,700 in total, and the recurrent weights are the adjacency matrix from above. In a normal RNN that matrix is dense, so every hidden unit reads every other unit at each step, and any input can reach any output in one step. Here each neuron reads only the roughly 150 neurons that connect into it, so information spreads one edge per token, and it is the fly's wiring that decides which neurons a token can reach and how many tokens it takes to get there.
Training is standard next-token prediction with backprop through the recurrence. The model has 27.6M trainable parameters, 25.6M of which are the synapse weights. That is about one-fifth of GPT-2 small.
Over the course of training, perplexity went from 4,096 to 15 on held-out programs. It picks the exact next token 45% of the time, and 75% of the functions it writes are valid Python.
In the video, every neuron is drawn at its real 3D position and colored by region (blue optic lobes, orange central brain, green nerve cord), and a sample of the synapses is drawn as faint lines between cell bodies.
Act I is training. Neurons light up as their synapses change, and the bright lines are the 500 synapses that moved most in the last 500 updates. Next to the brain, the same prompt is decoded at every checkpoint so you can watch the output improve, from random tokens at the start, to syntax-shaped nonsense, to a correct-looking loop by the end.
Act II is the trained brain writing is_prime from an initial prompt. Neurons light up as their state changes on each token, the bright lines are the signal in flight, and the video marks what is wrong with the result line by line.
Nothing about this brain evolved for code though.
It evolved for vision, flight, walking, smell and courtship, but it still learns Python, and the wiring itself is doing work. In fact, in a control experiment, if I keep every neuron's number of connections but shuffle who connects to whom, perplexity gets a third worse, 15.0 to 20.1. And if I freeze the synapses at their anatomical values and train only the output layer, it collapses to 41.9.
It is really amazing that evolution's connectivity helps it learn, even on a task evolution never saw or intended.
Imagine what a whole mouse brain, with hundreds of times more neurons, would be able to learn. Training on real wiring at that scale may teach us things about architecture that no search over transformer variants would find, and the end of that road may be running on the biological hardware itself instead of on a GPU.
What if pig farmers end up with more compute than Jensen 😂
2
1
19
1,778
We've added another set of frontier models to TogetherBench: GPT-6 Astra, Grok 4.6, Gemini 3.8 Flash, Claude Opus 5 and Opus 4.7.
In each evaluation dimension:
pass@1 / pass² — the bars. The full bar length is pass@1, the fraction of runs that fully solve the task (judge ≥ 0.85). The solid part is pass², the share of tasks solved in both runs. The hatched tail between them is instability, tasks the model solves only some of the time. Fable 5 leads both (70% / 62%); Grok 4.6 has the largest tail (59% / 44%): decent on a good day, but least repeatable.
Judge — an agentic judge (Claude Opus 4.6) scores each patch against weighted task-completion goals frozen per task, so partial credit is comparable across models.
Correction — how often the simulated user has to step in: # of corrections + 0.2 × # of nudges per task. Lower is better; it measures how much hand-holding a model needs, not just whether it gets there.
$ / task — new column. Average model API cost per task at each vendor's public list price, computed from actual token usage (uncached input, cached input, cache writes, output + reasoning). The most cost-efficient is Muse Spark 1.3 ($3.19), then Grok 4.6 ($3.64) and Gemini 3.8 Flash ($4.15).
Fortunately or unfortunately, there is no single winner across all dimensions, so the right model really depends on what you expect from it.
SWE-Together Update: Claude Fable 5 and 5.1 are the new top models on SWE-Together, and Meta's Muse Spark 1.3 is the best value on the board.
SWE-Together is our benchmark of 109 real coding tasks with a simulated user in the loop, built to tell you whether a new model will actually match your expectations before you hand it your work.
Six quick findings from the updated board:
1. Fable 5 beats 5.1 on consistency, but 5.1 is more efficient.
At their best the two actually perform the same. The gap is in the worst runs, where Fable 5.1 scored zero on 14 trials and Fable 5 on only 6. However, 5.1 is 25% faster, about 2x cheaper, and needs slightly fewer corrections from the user (1.45 vs 1.53 corrective messages per task) than Fable 5.
2. Muse Spark 1.3 is the value outlier.
Per solved task, it is about 2.5x cheaper than Fable 5.1 and 5x cheaper than Fable 5. Performance-wise, it ties with Fable 5 and 5.1 for best on Python (73%) and Rust (100%).
3. Newer is not automatically better at collaborative coding.
GPT-5.6-Sol is the newer model, yet it needs more corrections from the user than GPT-5.5 (1.66 vs 1.59 per task).
4. "Stronger models need less steering" is a trend, but not guaranteed.
With more models added, the correlation between pass@1 and user correction weakens from -0.92 to -0.72, and Fable 5 takes more corrections than Opus 4.8 (1.53 vs 1.38) despite a 6 pp higher pass@1.
5. Frontier progress contributes greatly to stability.
Fable 5 converts 89% of its pass@1 into pass² (both runs solve), versus 60 to 67% for DeepSeek V4 Pro, GLM-5.1, and MiniMax 2.7.
6. There is still plenty of headroom.
16 of 109 tasks have zero solves across all tested models, and even Fable 5 at 69% pass@1 sits about 9 pp below the ~78% that the original human patches scored.
The full leaderboard, with per-task and per-trial breakdowns, is at togetherbench.com, and the benchmark is open source at github.com/Togetherbench/SWE….
11
4
75
13,026
Zhuokai Zhao retweeted
There are a lot of things wrong with this world…
but too much intelligence is not one of them.
55
459
2,924
339,403
SWE-Together Update: Claude Fable 5 and 5.1 are the new top models on SWE-Together, and Meta's Muse Spark 1.3 is the best value on the board.
SWE-Together is our benchmark of 109 real coding tasks with a simulated user in the loop, built to tell you whether a new model will actually match your expectations before you hand it your work.
Six quick findings from the updated board:
1. Fable 5 beats 5.1 on consistency, but 5.1 is more efficient.
At their best the two actually perform the same. The gap is in the worst runs, where Fable 5.1 scored zero on 14 trials and Fable 5 on only 6. However, 5.1 is 25% faster, about 2x cheaper, and needs slightly fewer corrections from the user (1.45 vs 1.53 corrective messages per task) than Fable 5.
2. Muse Spark 1.3 is the value outlier.
Per solved task, it is about 2.5x cheaper than Fable 5.1 and 5x cheaper than Fable 5. Performance-wise, it ties with Fable 5 and 5.1 for best on Python (73%) and Rust (100%).
3. Newer is not automatically better at collaborative coding.
GPT-5.6-Sol is the newer model, yet it needs more corrections from the user than GPT-5.5 (1.66 vs 1.59 per task).
4. "Stronger models need less steering" is a trend, but not guaranteed.
With more models added, the correlation between pass@1 and user correction weakens from -0.92 to -0.72, and Fable 5 takes more corrections than Opus 4.8 (1.53 vs 1.38) despite a 6 pp higher pass@1.
5. Frontier progress contributes greatly to stability.
Fable 5 converts 89% of its pass@1 into pass² (both runs solve), versus 60 to 67% for DeepSeek V4 Pro, GLM-5.1, and MiniMax 2.7.
6. There is still plenty of headroom.
16 of 109 tasks have zero solves across all tested models, and even Fable 5 at 69% pass@1 sits about 9 pp below the ~78% that the original human patches scored.
The full leaderboard, with per-task and per-trial breakdowns, is at togetherbench.com, and the benchmark is open source at github.com/Togetherbench/SWE….
8
8
80
24,884
Suppose two students take an exam in the same course, except one gets a much harder version than the other. But both are graded against the same class average.
It is obvious that this grading is flawed. The student with the easy version can get a good grade for an ordinary understanding of the course, while the student with the hard version can get a bad grade despite actually understanding it better.
But this is exactly what happens to an RL agent trained across many levels of the same game with one shared critic. The levels are the exam versions and the critic is the class average. No teacher would grade this way, yet it is the default in nearly every parallel RL implementation.
That's why I'm excited about our new paper, which isolates this mechanism, proves what it does to the learning path, and proposes a fix that costs almost nothing.
The cleanest place to see the mechanism is the paper's CartPole experiment (balance a pole upright on a moving cart). Two levels differ only in gravity (e.g., 10 and 50), and gravity is hidden from the observation. The reward is survival time (one point for every time step the pole has not fallen yet).
Now put the agent in the same state-action scenario (same cart position, pole angle, push) but under two gravities. Gravity is what tips the pole over, so under gravity 50 it falls away from upright much faster and the same push buys far less correction.
The critic's job is to predict exactly this, how many more steps the pole will stay up from the current state, and every move gets judged against that prediction. The true answer is large under gravity 10 and small under gravity 50, but the shared critic sees the same state in both levels and has to output one number, roughly the average of the two.
In the light level that number is too low. A mediocre move still beats it, so the agent learns to repeat mediocre moves.
In the heavy level the number is too high. A good move falls short of it, so the agent learns to avoid good moves.
Averaged over both levels the two errors cancel, which is the sense in which the shared critic is "correct." But the agent never trains on the average. It trains on one episode at a time, and every episode comes from an agent that just absorbed the previous mistake. Errors that cancel on paper do not cancel along a run.
Nothing in this story is specific to gravity. The paper calls the gap between a level's true value and the shared average "value mismatch," and it appears whenever levels share the same visible state but differ in hidden dynamics or horizon.
The shared critic fits the average across levels because that is exactly what a critic without level information is asked to fit, so more data does not fix it, more training does not fix it, and GAE does not filter it out.
I am skipping the theory here, which proves in multi-environment bandits that the shared and per-environment baselines have identical expected updates and the same limit, yet only the per-environment one eventually turns every sampled update into a step forward while the shared one keeps stepping backward infinitely often.
The paper's fix is minimal: "Give only a logged environment index to the critic so that it can separate the value targets." The index is an arbitrary label that tells the critic which level it is in, which in the exam analogy means grading each version on its own curve.
But the results from this fix are phenomenal.
On BipedalWalker, mean return on 100 unseen terrains goes from 90.7 with the shared critic to 155.6 and 190.4 with the two conditioned critics, and on Procgen the conditioned critics beat the shared one in all 16 games, improving aggregate normalized return on unseen levels by 21.2% and 40.8%.
What I find interesting beyond the specific method is how directly this may apply to LLM agent RL, even though the paper never mentions an LLM.
For example, GRPO grades on a curve by construction. Sample a group of rollouts for one prompt, subtract the group mean, that is the curve. However, it is the right curve only if every rollout in the group faced the same exam.
In agentic RL they often do not. The same task gets instantiated with different sandbox seeds, hidden test subsets, repository states, or data draws, and if groups are keyed by task, the group mean is the shared critic. Rollouts on easy instances earn positive advantages for mediocre behavior, rollouts on hard instances earn negative advantages for good behavior. The direct analog of the fix is then keying groups by (task, environment config) instead of task.
However, the obvious cost is smaller groups per instance and therefore noisier curves, so it only pays off where instances actually differ, which a harness can check from per-instance pass rates before splitting.
Paper: arxiv.org/abs/2608.26481
Interactive demo: liu-zhenya.github.io/shared-…
5
5
35
3,676
many people say never bet against google after today's gemini 3.8 flash release
but they'll never catch up
8
1
19
12,088
This is crazy… direct 4D training at 1T context, inference past 5T, what?!
For reference, counting raw pixels or grid points:
- Video gen (Veo 3.1, Kling 3.0, Seedance 2.0) has ~1B per clip (8–15s, 1080p–4K). The whole clip is generated in one shot, but no explicit 3D.
- Marble is 3D but static (no time axis), built from a ~10⁷ px panorama and lifted into a persistent scene of roughly as many points.
- Genie 3 is interactive, 720p @ 24fps, generated frame by frame as you move. With ~1 min of memory that's around 1.3B pixels.
mind blown.
Excited to see @Reuters cover the launch of our startup Accelerated Understanding.
We are training large scale AI models that can simulate and understand physics to invent and discover. Our models understand the world directly in 4D (3D + time) and across physical phenomena. Going full 4D requires massive context length, we have pushed it to a Trillion in training and exceeding 5 Trillion at inference.
AI giving you a bigger haystack of ideas doesn’t help. The bottleneck for new inventions and discoveries is shifting from ideas to the ability to test them. With AI that can simulate and understand physics we are directly attacking this bottleneck.
People have been trying to do this for a while now, but usually by taking shortcuts. Narrow surrogates are great if you happen to have enough of precisely the right data and your design loop stays in distribution. Video models look fantastic but sweep physical accuracy under the rug, and some static world models cut out physics altogether. A lot of interesting physics isn’t visual.
What does not cutting corners look like? Space stays 3D and you also have time: so 4D in total. You also need multiple physical modalities in the same model, not just things you can see. That’s what we’ve built.
Scaling is the primary ingredient to make this work. To represent the world you need sufficient context, which in our case grows in 4 dimensions. Individual samples get so big they don’t fit into single accelerators or even full nodes anymore.
We’ve developed architectural tricks to make it work. We’ve pushed our models to 1T parameters during large scale pre-training and are able to train at up to a Trillion context when needed and do inference exceeding 5 Trillion context without any sub-sampling or patching.
Building on prior successes of AI weather forecasting, fusion simulation, design of medical devices, drugs and chips, we wanted to see if scale and universality can benefit AI for physical understanding. With our teams’ experience in large-scale infrastructure and model training we’ve been able to pull it off.
reuters.com/business/ai-foun…
acceleratedunderstanding.com…
@accelerated_u @bjenik
2
5
76
13,865
Pretty interesting rethinking paper on "discouraging" self-evolving loops.
So the background is that most current self-evolving loops run their search directly on the test set. It kind of makes sense as harness search needs accurate, verifiable feedback to make grounded edits.
However, that quietly turns self-evolving loops into a form of test-time scaling, which is exactly what this paper argues.
Specifically, the paper points out that a loop that repeatedly evaluates and revises candidates against task feedback, then reports on those same tasks, is logically a test-time search procedure. So its gains should be measured against test-time scaling under matched feedback and inference budgets — otherwise you can't tell whether it discovered a better harness or just spent more compute.
With that in mind, the paper runs four methods under the same budget:
1. parallel sampling — fixed harness, k independent trajectories per task, with a self-judge or unit tests picking the final answer
2. sequential refinement — fixed harness, k retries in a row. Each round summarizes the previous attempt into context and tries again (essentially prompt refinement)
3. harness evolution — the standard self-evolving loop. One shared harness, revised each round from feedback pooled across all tasks
4. harness scaling — the per-instance counterpart. Each task evolves its own harness
The results are very interesting.
Harness evolution doesn't beat plain parallel sampling, and without verifiable feedback it can even fall below single-attempt direct sampling with the initial harness.
Its gains also show up at pass@5 but barely at pass@1, which implies that the improvement comes from taking multiple attempts, not from the harness getting better.
And on a disjoint search/eval split the evolved harness transfers almost nothing to held-out tasks, which means the edits memorize task-specific fixes rather than distill reusable strategies.
There is one caveat, which is that the "unified budget" only counts inference on the tasks, not the compute spent generating harnesses. But this flaw kind of favors harness evolution, and it already loses.
So in my opinion this really shows that existing self-evolving loops might just be a different way of applying test-time scaling, rather than some new intelligence discovery.
And we should focus more on making generalizable self-evolving loops work!
15
40
249
20,622
amazing launch, best-in-class agentic performance with additional support for fast local inference, and
guess @denny_zhou is right --- data and scaling are what remain for now, if we trust Transformer+Reasoning is AGI.
Today we're also opening the weights for Muse Glimmer, a great 30B parameter dense model that can run locally. Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model. Meta is a strong supporter of open source and I'm proud of these releases. Congrats to @alexandr_wang and the MSL team for all your great work on these models.
2
2
23
2,914
actually bought two after seeing the post 😂
This was one of the coolest things I got for my kids. Thanks for the recommendation @realcezarc
Stickerbox!
7
1,194
Is it just me or does the Claude mac app really randomly lose recent chats every time it updates?
2
1
12
2,144
this touches a deeper aspect of current AI research: if the same core theory is applied to study different questions, does that count as plagiarism?
I'd probably say no, especially in this case.
The XM paper cites IMLE right before introducing its core objective (Eq. 1), and Appendix E.3 argues that IMLE is a specific instance of end-to-end Forward XM. It also pushes back on IMLE's theory, arguing the working mechanism was never implicit maximum likelihood but the multi-candidate search itself.
In this case I think novelty (or contribution) lives more in the question, not just the method.
IMLE asked how to avoid mode collapse in conditional image synthesis. XM asks whether the same best-of-K objective (sample K candidates, backprop only through the one closest to the data) works as a third pre-training axis, with gains that grow with scale rather than saturate. The IMLE line of work never pursued these questions.
Probably @AlexiGlad should have called out IMLE more in Section 3 and said plainly that Eq. 1 is the conditional IMLE objective (which I personally would also find it a bit odd). But this does not make it plagiarism.
If reusing a core mechanism to answer new questions were plagiarism, much of modern ML would be guilty.
2
6
68
14,857