Glad to share that 3 papers I contributed to at @ScaleAILabs have been accepted to NeurIPS 2026, spanning post-training and agent evaluation 🎉 🔍 Reward Hacking in Rubric-Based Reinforcement Learning We conducted a systematic study of reward hacking in rubric-based RL, distinguishing failures caused by weak verifiers from failures in rubric design, and identified an interesting stopping criterion. arxiv.org/abs/2605.12474 🧠 Not Every Rubric Teaches Equally We introduce a policy-aware curriculum for rubric-based RL that emphasizes the criteria most useful for learning at each stage of training. arxiv.org/abs/2605.20164 🛠️ SWE Atlas We introduce a benchmark for evaluating coding agents beyond issue resolution, covering codebase Q&A, test writing, and refactoring. arxiv.org/abs/2605.08366 Really proud of the team and the work behind these. Looking forward to sharing more and catching up with everyone at NeurIPS in Sydney!

Excited to share three @ScaleAILabs papers accepted to NeurIPS 2026 in the Evals and Datasets track! SWE Atlas: Evaluating coding agents beyond issue resolution, across codebase Q&A, test writing, and refactoring—with attention to both correctness and engineering quality. arxiv.org/abs/2605.08366v1 MCP-Atlas: Evaluating agents’ ability to discover and use tools across real MCP servers to complete realistic, multi-step tasks. arxiv.org/abs/2602.00933 Beyond Truthfulness (MASK): Separating honesty from accuracy by testing whether models contradict their own stated beliefs when pressured to lie. arxiv.org/abs/2503.03750v2 Agent evals and benchmarks are important as capabilities increase, safety becomes critical, and reliability remains an open question. Excited to continue this work with our customers, and grateful to all my co-authors and collaborators for the hard work behind these benchmarks!

"MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers" has been accepted to NeurIPS 2026!! I had the privilege to be co-first author on the paper and to work with virtually all of the frontier labs to enable them to report numbers on our benchmark!

🎉 Excited to share that “Reward Hacking in Rubric-Based Reinforcement Learning” has been accepted to #NeurIPS2026! Reward hacking in post-training is a particularly interesting and important problem to work on. We study how models can learn to game rubric-based rewards during post-training, satisfying the rubric without necessarily improving the broader response quality. We characterize these failures at the criterion level and show that stronger verifiers help, but don’t fully eliminate reward hacking. Link: arxiv.org/abs/2605.12474 Check out some of our recent post-training research projects at @ScaleAILabs: RGSD: arxiv.org/html/2606.12507 POW3R: arxiv.org/abs/2605.20164

🎉 Excited that “Not Every Rubric Teaches Equally” is accepted to NeurIPS 2026! 🎉 POW3R turns static rubric rewards into policy-aware training signals, emphasizing the criteria that actually separate good and bad rollouts for the current state of the model. Link - arxiv.org/pdf/2605.20164 Our team at @ScaleAILabs has been cooking on some really exciting RL/post-training research lately. Check it out 👀 RGSD - arxiv.org/abs/2606.12507 Reward Hacking with Rubrics - arxiv.org/abs/2605.12474

600+ proposals in 🤯 We're reaching out to top contributors to start building in our public repo. We’ve also welcomed @SchmidhuberAI, a pioneer of RSI, as a senior advisor to RSI Bench. New blog on our setup and verification pipeline: rsi-benchmark.com/blog/verif…

AI safety doesn’t translate one-to-one. We partnered with the Korea AI Safety Institute to develop ROK-FORTRESS, our first benchmark together, testing how language and geopolitical context affect AI safety. Across 14 frontier models, we found that translation alone can miss meaningful differences in model behavior. scale.com/blog/korea-ai-safe…

New research from Scale Labs: SteerDuplex and SteerBench measure how well voice models follow instructions for how a response should sound and be delivered. +44.5 points on audio steering over the best open baseline. Pause barge-ins cut from 26.5% to 9%. Plus a sharp result on why timing rewards alone collapse into silence. ⬇️

GPT-6 Astra is a step function on DrugDiscoveryBench at 68.7%, our benchmark for early-stage drug discovery computational workflows. Muse Spark 1.3 and Opus 5 follow. @TuXinming, @KavanaghOscar and I will talk about what it measures tomorrow at 10 am PT! Register below ⬇️

JUST ADDED: @OpenAI GPT-6 Astra, @AnthropicAI Claude Fable 5.1, and @GoogleDeepMind Gemini 3.8 Flash just joined our leaderboards. Check out the updated rankings: labs.scale.com/leaderboard

New models this week are much more codebase-aware: they check and break their own tests, and like senior/experienced engineers, they even care about code hygiene. As part of that, I want to shout out some major gains we saw on our SWE Atlas leaderboard @ScaleAILabs. @AnthropicAI is tied for 1st on all three SWE Atlas leaderboards, with the latest Opus and Fable models clustered at the top. But Fable 5.1 feels like a qualitatively different model: more consistent, more efficient, and noticeably more codebase-aware. In Test Writing, it's the first model we've seen that actively tries to mutate code and break its own tests to check for robustness. In Refactoring, it consistently cleans up old leftover code without being asked. On SWE Atlas, no model has cracked 70 percent yet. There's a lot of work these agents still can't do.