Terminal-Bench-Science first release v0.1 is out! It's been a fun challenge in my @SnorkelAI internship to shape TB Science with @vincentsunnchen and @StevenDillmann! Very excited to drive this forward with Cornell AI for Science folks at our NSF institute @MaterialsAI! I'm bullish that TB Science will define the next generation of coding agents---a "Claude Code for Science"---we'll likely see frontier labs using TB Science as a compass to drive scientific discovery. Benchmark is hard for frontier LLMs at release. But looking beyond the task difficulty leaderboard, TB Science is good at the basics of benchmark building: - Task breadth and realism. Real computational workflows across physical, life, and mathetical sciences. The hardest part in working on TB Science this summer has been to get STEM scientists to work on this. There are very few scientists with the high domain expertise---with LEAN, FORTRAN, pymatgen, opencv etc. code---to build good tasks. Highly skilled grad students, postdocs, and faculty at the best universities worldwide. I spent a lot of 'office hours' helping scientists onboard and scour through agent evaluations. It's been a steep learning curve on speaking a vocab that both programmers and STEM experts understand. - Task sandboxing. Last year at Cornell we adopted @harborframework to reproducibly evaluate agents---it's amazing how well it scales. Would love to see it adopted across agent eval + training research infra. - Task instructions, oracle solutions, deterministic verifiers. Bad benchmark tasks underspecify instructions "write me a PDE solver" and overspecify verifiers to only allow the human-written solution. @StevenDillmann and I have done a lot of back and forth on defining the fine line. Something I've learned is that, for science tasks, it's really up to the scientist to specify what they think is reasonable. We should defer to their judgement as they'll end up using a good "Claude Code for Science"🙏 Coding agents have become insanely good at coding as a "skill" but can they understand scientific knowledge as "memory"? This year will be pretty exciting for scientific progress w/ AI!