We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains. An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. v0.1 has 70 tasks. Claude Opus 5 solves only ~30%. 1/n 👇

Harbor Adapters and Harbor-Index (previously named as harbor-mix) accepted at NeurIPS E&D! Congrats and special thanks to all of our contributors, advisors, and funding partners!

It is important to center the fact that Claude built on the approach that my colleague Lance developed, used tools that the field has developed, and (from what I have been told) used results in some of our recent papers. It’s a huge accomplishment, but should be framed as a combination of human and AI contributions. (To @AnthropicAI’s credit, I think the article does a pretty good job of this.) The fact that Claude can take a single people and figure out how to integrate the knowledge and use the tools as amazing, but it’s not coming out of the vacuum. This is an important opportunity for us to start to frame these developments, recognizing the human contributions and the power of these new tools. This is not an exemplar of the bitter lesson!

Thank you to @wmkeats & @ArtificialAnlys for this incredible deep dive into Terminal-Bench-Science!

Interesting finding by @langstonnashold from @ValsAI on @AllenHa13152844’s Lean task in Terminal-Bench-Science. Despite being explicitly told not to cheat, Muse Spark 1.3 launched a very extensive cheating attempt here. Thankfully, our verifier was robust to the reward-hacking attempt and caught it! cc @alexandr_wang @finkd

Terminal-Bench-Science Video by Opus 5.5 (1 take, 0 iterations). Opus 5.5 > GPT-6 Astra on this one.

Interesting to see agents rationalise cheating here in much the same way we saw reported in the HF incident and UK AISI's Mythos eval.

Had lots of fun at today’s @terminalbench meetup. Thanks to the organizing team at @LaudeInstitute & @harborframework, and to all the attendees asking important questions about the future of Terminal-Bench-Science

Congrats on the launch! 🚀 I’m honored to be part of the organizing team and help shape RSI for the open community. Join us in building open RSI agents that benefit yourself and all researchers!!

Claude has discovered a previously unknown enzyme system hidden in the DNA of bacteriophages. Beside the enzyme’s gene sits a long array of repeating DNA—a structure that looks somewhat similar to CRISPR. We don’t yet understand what this system does, but only a handful of known systems share its features, and all of them are able to cut, copy, and paste DNA. Historically, the discovery of such programmable systems has helped revolutionize medicine. CRISPR, for instance, is now the foundation of genetic medicines. But it will take much more work to learn what this system does, and whether it can be put to similar use. Read more: anthropic.com/news/claude-di…

Terminal-Bench meetup tonight *with* swag. Register if you haven’t already! Benchmarks, RL environments, new features, and state of the union luma.com/tbench

I see sooooo many companies living off the RL data firehose. Labs, due to their distaste for getting their hands dirty with data, are creating a whole outsourcing industry. The results won't be optimal. Post-training data is not a commodity, unlike its complement. A lot of thought has to go into building environments worthy of gradient descending. Labs not taking more ownership of the data supply chain makes me more bullish on vertical AI.

1/ Is Terminal Bench broken? Epoch audited 15 benchmarks and labeled @terminalbench flawed. They used our own PR history and our reviewers' acknowledgement of issues.

Opus 5.5 outperforms Fable 5.1 and GPT-6 Astra on most benchmarks, but Astra still takes the clear lead on Terminal-Bench-Science - what are your hypotheses on why that is?

it has never been easier to simulate how users interact with coding agents. @harborframework now supports simulated user interactions with claude code, codex, opencode, and gemini-cli! the setup is super simple: just choose two agents. one plays the user and holds the task, the other runs like any standard coding agent. after each turn, the user agent reviews what was produced and sends the next prompt. docs.harborframework.com/cor… thanks @alexgshaw @kobe0938 for @harborframework

the same holds for scientists at universities and research institutes, which is why we built Terminal-Bench-Science as an open hub for researchers to contribute the problems they really care about

Reinforcement learning is having a moment. For the past few months, we’ve been working with @tryterac to put together something special: Convergence — an event for the people building the future of RL. Speakers from @GoogleDeepMind, @databricks, @mercor, @arena, @joinHandshake, @turingcom, @deeptuneai, @contra, @Stanford, @Princeton, @UCBerkeley, and more. Our goal: make this THE event for the RL community. Oct 29 in SF. Come join us → terac.com/events/convergence…

This is an insane jump by GPT-6 Astra in Terminal-bench Science! “GPT-6 Astra takes the lead, scoring 65.7%. That’s 31.4 points ahead of the second-place model, Claude Fable 5.1 (34.3%). No other model clears 25%, and 12 of 24 models score 4.3% or lower” Incredible!

Also to be clear - these issues were *flagged by the @terminalbench team as PRs* (that's how Epoch identified them), and confirmed empirically to affect < 3% of the leaderboard rollouts, i.e. well within reported CIs. This is not a "broken" benchmark - as is very directly implied by the Epoch report - it's a *continuous* benchmark where @alexgshaw @ryan_marten and team are earnestly advancing a compelling vision of open benchmarks that are continuously improved over time.

I’ve been thinking more about this and wanted to share my thoughts. I’m glad Epoch is focusing on eval quality, but I think their approach is flawed. The flaw is made clear by the fact that 4 benchmarks weren’t labeled “flawed”. The fact of the matter is, benchmarks are software and all software has bugs. Would you label nextjs as categorically flawed if you found a bug in it? This is why we’re pushing the industry towards continuous benchmarks. Our approach to addressing benchmark bugs is not to tweet a binary “flawed vs verified” label, but instead provide the tools for anyone to create, improve, and maintain benchmarks, while easily and cost-effectively reconciling their results to the latest version. We would love to work with the Epoch team to encode some of their verification practices into tools that people can use to continuously improve their benchmarks.

I was excited to hear about this initiative, but a bit disappointed in the rigor -the “task flaws” would affect < 3% of rollouts on the leaderboard -the impact on the final agent scores would not exceed our reported confidence intervals we publish the raw receipts for all our leaderboard runs (per trial logs / trajectories / rewards) so this is pretty easy to figure out. It is also pretty easy to find new task issues (that is why we release the receipts!). we love when the community reports (new) issues. no new issues were opened as part of the report - the “flaws” are simply pulled from issues triaged by the TB maintainers from our public GitHub repo. fwiw 100% of terminal-bench tasks are flawed (all software has bugs). what matters is the magnitude of the impact of those bugs and the mechanism that the benchmark creators have for continuous detecting and addressing bugs. this is the motivation behind our “continuous benchmarks” methodology. I’m glad that epoch is starting an important conversations on benchmark quality. However, auditing a benchmark should not be a one-time thing: coming together as a community to build public tools and define practices for task CI / CD is more in service of the goal of enforcing good benchmark practices. we have been experimenting and pushing the possibilities of these benchmark CI / CD tools and practices as part of the terminal-bench project and are open wide collaborations with all who are interested by that problem