Research Scientist @reflection_ai. Research on Reinforcement Learning, Agents, Reasoning. Ex: @allen_ai
- Tweets 168
- Following 866
- Followers 453
- Likes 914
Check out @bhutanisanyam1’s analysis on harnesses in agentic evals!
How does a harness affect LLM Behaviour?
Frontier models perform really well on benchmarks. So we measured how a harness can affect their behaviour:
- We run GLM-5.3, Inkling and Kimi-K3 on our release
- We run the eval through mini, Pi and OpenCode and measure a few things
- First-models generalise really well and the differences exist at the token level
- Kimi is the largest model but most token efficient when paired with Pi
- Pick your combination carefully for any model
- Failure on a task counterintuitively happens not because a model is lazy. Infact the models spend twice as much compute and steps, they try harder
- Each harness gives different tools, how the models pick them varies. GLM prefers testing the most, Inkling spends most of its time reading files.
It was a huge honor to be part of @reflection_ai x @scale_AI release. I learned a lot 🙏
labs.scale.com/blog/swe-benc…
2
287
Jeff Da retweeted
How does a harness affect LLM Behaviour?
Frontier models perform really well on benchmarks. So we measured how a harness can affect their behaviour:
- We run GLM-5.3, Inkling and Kimi-K3 on our release
- We run the eval through mini, Pi and OpenCode and measure a few things
- First-models generalise really well and the differences exist at the token level
- Kimi is the largest model but most token efficient when paired with Pi
- Pick your combination carefully for any model
- Failure on a task counterintuitively happens not because a model is lazy. Infact the models spend twice as much compute and steps, they try harder
- Each harness gives different tools, how the models pick them varies. GLM prefers testing the most, Inkling spends most of its time reading files.
It was a huge honor to be part of @reflection_ai x @scale_AI release. I learned a lot 🙏
labs.scale.com/blog/swe-benc…
6
7
52
5,503
We're releasing SWE-Bench Pro v2. The dataset is rebuilt from the ground up, and all task instructions, verifiers, and images went through multiple rounds of expert review. Give it a try!
Today, we released SWE-Bench Pro V2, a refreshed public split of SWE-Bench Pro.
The frontier keeps moving, and the standard for measuring it should move with it. We’ll keep strengthening our leaderboards, incorporating feedback, and building more rigorous evals for what comes next.
1
1
8
735
Jeff Da retweeted
We are sharing an early preview of our ongoing SWE-1.6 training run.
It significantly improves upon SWE-1.5 while being post-trained on the same pre-trained model - and it runs equally as fast at 950 tok/s. On SWE-Bench Pro it exceeds top open-source models.
The preview model still exhibits some undesirable behaviors like overthinking and excessive self-verification, which we aim to improve. We are rolling out early access to a small subset of users in Windsurf.
62
108
1,207
550,556
OpenAI is moving away from SWE-Bench Verified, citing challenges on underspecified tasks, misaligned tests, and contamination.
We agree. These were exactly the motivations behind SWE-Bench Pro (arxiv.org/pdf/2509.16941).
What we changed:
→ Underspecified tasks: structured, executable problem definitions
→ Contamination: strict curation + private / commercial codebases
But this is just step one.
Where we’re pushing frontier coding evals next:
→ Beyond unit tests: rubric-based evaluation (arxiv.org/pdf/2601.04171)
→ From static tasks to real-world agentic environments
Modern coding systems are not solving isolated problems. They operate as agents over repos, tools, and long-horizon workflows. Our evals need to reflect that.
SWE-Bench Pro is one step toward more realistic and reliable evaluation for coding agents.
We’ll keep pushing the frontier.
2
3
38
3,384
Jeff Da retweeted
The standard for frontier coding evals is changing with model maturity.
We now recommend reporting SWE-bench Pro and are sharing more detail on why we’re no longer reporting SWE-bench Verified as we work with the industry to establish stronger coding eval standards.
SWE-bench Verified was a strong benchmark, but we’ve found evidence it is now saturated due to test-design issues and contamination from public repositories.
openai.com/index/why-we-no-l…
95
128
1,475
248,825
Jeff Da retweeted
Introducing Gemini 3.1 Pro, our new SOTA model across most reasoning, coding, and stem use cases!
546
562
7,371
649,097
Jeff Da retweeted
Introducing M2.5, an open-source frontier model designed for real-world productivity.
- SOTA performance at coding (SWE-Bench Verified 80.2%), search (BrowseComp 76.3%), agentic tool-calling (BFCL 76.8%) & office work.
- Optimized for efficient execution, 37% faster at complex tasks.
- At $1 per hour with 100 tps, infinite scaling of long-horizon agents now economically possible
MiniMax Agent: agent.minimax.io
API: platform.minimax.io
CodingPlan: platform.minimax.io/subscrib…
445
1,002
8,531
5,364,729
Jeff Da retweeted
GPT-5.3-Codex's much better token efficiency *AND* faster inference is the biggest story of this release. Folks at @OpenAI worked hard to improve this and it will only get better from here.
GPT-5.3-Codex is here!
*Best coding performance (57% SWE-Bench Pro, 76% TerminalBench 2.0, 64% OSWorld).
*Mid-task steerability and live updates during tasks.
*Faster! Less than half the tokens of 5.2-Codex for same tasks, and >25% faster per token!
*Good computer use.
35
87
1,031
156,959
Jeff Da retweeted
GPT-5.3-Codex is here!
*Best coding performance (57% SWE-Bench Pro, 76% TerminalBench 2.0, 64% OSWorld).
*Mid-task steerability and live updates during tasks.
*Faster! Less than half the tokens of 5.2-Codex for same tasks, and >25% faster per token!
*Good computer use.
1,503
1,545
19,188
2,387,763
Jeff Da retweeted
This release is an emtional one for me because I had stayed up so much for it 🥹 It has been truly amazing to see this model becomes better bit by bit through every change we make, and we have come a long way.
Since I did mid-training for this model, I wanted to share a little anecdote about this part. We really made this model with user experience as first-class consideration. We want people to actually use it, period. We took it so serious that we redid midtraining because we saw cases where models failed to follow instructions on out-of-distribution scaffolds. We decided straight-up that we would fix this in a fundamental way instead of surface-level patching. The resulting base model, which we also release, is thus a healthy base. We find that, compared to other base models, this one better learns new tasks.
Try fine-tuning our base and lmk what you think 🥳
huggingface.co/Qwen/Qwen3-Co…
🚀 Introducing Qwen3-Coder-Next, an open-weight LM built for coding agents & local development.
What’s new:
🤖 Scaling agentic training: 800K verifiable tasks + executable envs
📈 Efficiency–Performance Tradeoff: achieves strong results on SWE-Bench Pro with 80B total params and 3B active
✨ Supports OpenClaw, Qwen Code, Claude Code, web dev, browser use, Cline, etc
🤗 Hugging Face: huggingface.co/collections/Q…
🤖 ModelScope: modelscope.cn/collections/Qw…
📝 Blog: qwen.ai/blog?id=qwen3-coder-…
📄 Tech report: github.com/QwenLM/Qwen3-Code…
55
82
1,352
109,316
Jeff Da retweeted
#1 open source on SWE-Bench Pro.
Ahead of Gemini 3 Flash. Level with Haiku 4.5.
Thanks @scale_AI for the solid benchmark. Let's keep pushing forward 💪
JUST ADDED: @MiniMax_AI 2.1 just joined our SWE-Bench Pro leaderboard. Check out the updated rankings: scale.com/leaderboard/swe_be…
8
11
231
20,698
Rubrics are effective verifiers for SWE-Agents!
🚀New @scale_AI research:
Verifiers for SWE Agents have traditionally used unit tests or simple, execution-free classifiers. But can we get verifiers that are more expressive, repository-grounded, and still execution-free at scoring time?
We explore Agentic Rubrics to fill this gap 💡
Agentic Rubrics are repo-grounded, execution-free verifiers for SWE agents. We generate a checklist of concrete, codebase-specific criteria using an Agentic Harness, and then score patches against it. 🧑💻
1
237
Jeff Da retweeted
🚀New @scale_AI research:
Verifiers for SWE Agents have traditionally used unit tests or simple, execution-free classifiers. But can we get verifiers that are more expressive, repository-grounded, and still execution-free at scoring time?
We explore Agentic Rubrics to fill this gap 💡
Agentic Rubrics are repo-grounded, execution-free verifiers for SWE agents. We generate a checklist of concrete, codebase-specific criteria using an Agentic Harness, and then score patches against it. 🧑💻
1
10
31
4,588
Jeff Da retweeted
Results:
- self-improvement on SWE-bench Verified (+10.4) and Pro (+7.8)
- better than the baseline RL using human issue data over the course of training
2
3
55
7,459
Jeff Da retweeted
New Scale research: Do AI models actually reason in ways humans can trust for real-world decisions?
Introducing MoReBench, the first benchmark for procedural moral reasoning in LLMs, measuring not just what models decide, but how they reason through moral ambiguity.
7
14
47
11,920
New open-source benchmark from @scale_AI: MCP-Atlas
MCP-Atlas is a large-scale benchmark for evaluating tool-use competency using 36 real MCP servers and 220 tools. The benchmark was featured in recent model cards (GPT, Claude, Gemini), and now it's open-source!
🚀 Today we’re open-sourcing MCP Atlas — a large-scale, real-server benchmark for agentic tool use, which has been used in the recent GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash model releases!
🧠 Key insight: realistic agentic tool use is not a function-calling problem. It requires tool discovery, orchestration, and recovery in real environments.
🔧 MCP Atlas evaluates agents on real MCP servers (36 servers, 220 tools, 1K human-written tasks). Models must find the right tools, call them correctly, chain them together, and handle failures.
📉 What we found:
• Agents fail more often at tool interaction than at reasoning
• Performance drops sharply with real-world tool friction
• Scaling models helps unevenly, robustness remains hard
• Claims-based eval reveals how agents fail, not just if they finish
Check it out!
📄 Paper: static.scale.com/uploads/674…
🌍 Environment: github.com/scaleapi/mcp-atla…
📂 Dataset: huggingface.co/datasets/Scal…
📊 Leaderboard: scale.com/leaderboard/mcp_at…
#AgenticAI #ToolUse #LLMEval #Benchmarks #MCP
1
3
296
Check out the paper and dataset:
Paper: static.scale.com/uploads/674…
Github: github.com/scaleapi/mcp-atla…
Dataset: huggingface.co/datasets/Scal…
Leaderboard: scale.com/leaderboard/mcp_at…
96
Jeff Da retweeted
We recently introduced MCP-Atlas, a benchmark for evaluating how well LLMs handle tool use via the Model Context Protocol. Even top models failed nearly half of realistic multi-tool tasks.
Today, we’re open-sourcing the benchmark so you can measure performance yourself.
1
6
32
4,965
Jeff Da retweeted
🚀 Today we’re open-sourcing MCP Atlas — a large-scale, real-server benchmark for agentic tool use, which has been used in the recent GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash model releases!
🧠 Key insight: realistic agentic tool use is not a function-calling problem. It requires tool discovery, orchestration, and recovery in real environments.
🔧 MCP Atlas evaluates agents on real MCP servers (36 servers, 220 tools, 1K human-written tasks). Models must find the right tools, call them correctly, chain them together, and handle failures.
📉 What we found:
• Agents fail more often at tool interaction than at reasoning
• Performance drops sharply with real-world tool friction
• Scaling models helps unevenly, robustness remains hard
• Claims-based eval reveals how agents fail, not just if they finish
Check it out!
📄 Paper: static.scale.com/uploads/674…
🌍 Environment: github.com/scaleapi/mcp-atla…
📂 Dataset: huggingface.co/datasets/Scal…
📊 Leaderboard: scale.com/leaderboard/mcp_at…
#AgenticAI #ToolUse #LLMEval #Benchmarks #MCP
12
20
217
21,972