San Francisco, CA
- Tweets 161
- Following 3
- Followers 2,241
- Likes 198
Harbor Framework retweeted
Terminal-Bench meetup tonight *with* swag. Register if you haven’t already! Benchmarks, RL environments, new features, and state of the union
luma.com/tbench
17
10
115
11,216
Harbor Framework retweeted
The gold standard for science is sharing reproducible and auditable experimental data.
When it comes to benchmarking agents, this means configs, trajectories, rewards, and logs for every single trial.
We built harbor to make this easy.
Unreal's analysis (related to @melissapan's recent work on comparing cost between harnesses while keeping the model fixed) sets a good example: they have uploaded and attached the raw harbor jobs for every result (each can be reproduced with a single harbor command). Good work!
Introducing Unreal Agent:
An open-source harness with state-of-the-art cost efficiency
39% cheaper than Codex+Astra on Terminal-Bench 4.0 while maintaining performance
7
3
80
8,678
Harbor Framework retweeted
Half the set is public with every judge's full logs, a scorer for your own judge, and the audit trail. 35 items are held out. Every case is a runnable @harborframework task, so you can point your own judge at the set and score it with one command.
Full writeup: heuristic.build/blog/arbiter…
1
8
316
Harbor Framework retweeted
@vercel’s fx.sh harness is now available as an agent judge in Harbor Rewardkit. It makes judge setup time 7× faster than Codex and 16× faster than Claude Code. @harborframework
1
2
6
455
Jev-as-a-Judge in Harbor Rewardkit!
Jev-as-a-Judge in Harbor Rewardkit is pretty amazing. It agrees with Fable and GPT and is very fast. When we test it on a real eval that uses an LLM judge, such as the @harvey LAB benchmark, Jev achieves perfect agreement but is 50x faster.
1
1
32
2,894
Harbor Framework retweeted
We love @harborframework's new observability feature!
Got streaming working with @tensorlake sandboxes on Harbor too. So you can see the agent's trajectory and files while it runs. PR is on the way.
1
12
1,760
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚑𝚊𝚗𝚍𝚜𝚑𝚊𝚔𝚎-𝚊𝚒/𝚊𝚝𝚕𝚊𝚜-𝚏𝚒𝚗𝚊𝚗𝚌𝚎
In my view, ATLAS Finance is the closest benchmark yet to real end-to-end professional financial work.
Code to run the benchmark as a @harborframework task suite: github.com/Handshake-AI-Rese…
Data/environments: huggingface.co/datasets/hand…
8
35
4,636
docs drop!
New Harbor docs!
We've released many exciting features that were previously undocumented like simulating users, streaming, and regrading trials.
Check them out or give your coding agent the MCP.
docs.harborframework.com/
9
1,785
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚋𝚎𝚜𝚙𝚘𝚔𝚎𝚕𝚊𝚋𝚜/𝚊𝚞𝚝𝚘𝚛𝚎𝚜𝚎𝚊𝚛𝚌𝚑-𝚎𝚡𝚊𝚖
We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability.
benchmarks.bespokelabs.ai/au…
In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve.
We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours.
Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets.
Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)
3
3
56
6,382
Harbor Framework retweeted
Add skills to your agent, swap the model, tweak a tool description. How do you know you didn't break what already worked?
Managed Deep Agents ships with @harborframework.
Evals run in a fresh container every time, tracked in LangSmith.
What you need to know in 9 minutes:
8
7
46
9,466
Harbor Framework retweeted
Registered - we have several ambitious new evals we’re kicking off in collaboration w/ some exciting names, and are deeply evaluating @harborframework as the foundation. Would be great to meet the team!
1
4
168
Harbor Framework retweeted
We're hosting a meetup!
Come meet the community behind the benchmark, celebrate recent releases, and join our discussions about the future of agent evaluation.
luma.com/tbench
3
5
54
20,585
Harbor Framework retweeted
Harbor Adapters and Harbor-Index
arxiv.org/pdf/2609.04298
1+year work, 120+contributors, 300+PRs, 10+ funding partners.
1
41
183
30,623
SkyRL + Harbor RL recipe!
Worked with Mercor Research and the SkyRL team, training Qwen3.5-397B-A17B on APEX-Agents (long-horizon office work) off-the-shelf data with SkyRL, improving Pass@1 from 16% to 27%.
The post is more of a practical field guide for what to do given an RL dataset, de-risking step by step: environment infra, harness fixes/optimizations, token-in-token-out, an overfitting run, small-model ablations, then the hero run.
We were also delighted to see the gains transfer to OpenCode despite training on an MCP-based harness, and even to Terminal-Bench 2.1.
Code, models, and eval traces are all open source. Hope it helps your next hillclimb!
4
62
7,812
Harbor Framework retweeted
Terminal-Bench-Science 0.1 is the #1 featured benchmark on @AnthropicAI’s new Claude Fable release 🚀
terminal-bench-science.ai
Across our benchmarks, the model sets a new standard.
It scores 52.6% on Terminal-Bench-Science 0.1, more than double Fable 5. On Terminal-Bench 4.0, it scores 55.8% against 42.0% for Fable 5.
ALT Benchmark table comparing Claude Fable 5.1 with Fable 5, Opus 5, and GPT-5.6 Sol across seven evaluations. Fable 5.1 leads on every row, including 52.6% on Terminal-Bench-Science 0.1 and 55.8% on Terminal-Bench 4.0.
13
20
136
8,965
Harbor Framework retweeted
We use @harborframework internally and have worked with @kobe0938 and the rest of the team to add support for simulated-user interactive benchmarks. This allows the community to more realistically model long sessions and cache expiration when evaluating coding agent and routing economics: github.com/harbor-framework/…
2
4
22
3,526
Harbor Framework retweeted
This is what powers the dynamic network policy support we added to @harborframework , allowing agents to switch network policies without restarting the running environment.
We shipped a comprehensive update to Tensorlake's egress traffic and firewall infrastructure for sandboxes.
Users can now update almost all aspects of traffic control while a sandbox is running. This lets a sandbox start up, download or configure tools, and then lock down before agents begin working in it.
Support for DNS-based policies improved as well:
1. Allow and deny based on DNS names (including wildcards).
2. Evaluate DNS-based flows per connection, so destination IP churn doesn't let traffic slip to denied endpoints.
2
1
447
Braintrust 🤝 Harbor
Some agents need a sandbox where they can do work, liking editing files, installing dependencies, and launching builds. To score these agents you run the tests and check what changed, which takes a clean container per attempt.
Harbor is a Python framework for specifying sandboxed agent tasks and running them in isolated Docker containers. Each task holds an instruction for the agent and a verifier that inspects the container once the agent stops.
With the Harbor native Braintrust plugin, you can eval an agent in a sandbox, compare across runs, and share with anyone, without moving a folder.
Read more → braintrustdata.link/harbor-a…
1
15
3,315
Harbor Framework retweeted
this is truly exceptional work. i had the chance to talk to Steven a bit about how he went about designing and building TB-Science, and i came away really impressed
making benchmarks is easy. making an outstanding one is unbelievably hard. the incentives are stacked against it, from diluting task quality to ship fast, to building hill-climbable problem shapes you can beat by buying envs on the market
see, many folks think of benchmarks as marketing tools, and in a way they are! but the best ones are way more. to me they're basically the AI version of GDP. heavily gamed and debated, yet still indispensable. a good public eval doesn't need to be ungameable. it needs to be hard enough to game that gaming it looks a lot like actually getting better
steven and his team built something special here. measuring the frontier of science and how fast (or how slow!) models are pushing it further is arguably one of the most important things in the world today. doing it with relentless rigor makes it a public good
can't wait to see TBScience v02 come out, and if you're a scientist, you should consider contributing!
We're releasing Terminal-Bench-Science: a benchmark for evaluating AI agents on research workflows across scientific domains.
An ongoing Stanford-led community effort, built by the team behind Terminal-Bench together with scientific domain experts at research institutions worldwide. v0.1 has 70 tasks. Claude Opus 5 solves only ~30%.
1/n 👇
1
4
42
3,726