echen retweeted
What does filling out a spreadsheet have to do with fixing a PowerShell parser?
More than we expected.
We post-trained a model on long-horizon office work (documents, spreadsheets, research, planning, tool use). Zero coding tasks.
It still improved +5.7pp on SWE-Bench Pro.
Looking at the trajectories, the model hadn’t learned more software engineering. It had gotten better at something more general: forming the right goals, building an accurate picture of its environment, keeping the parent objective intact, and checking that the job was actually done.
We call this Goal-Directed Execution.
The takeaway: well-designed data can teach general capabilities, not just domain knowledge.
Learn more:
surgehq.ai/blog/office-work-…
2
2
7
671
echen retweeted
We trained Qwen3-4B on 1,000 ComplexConstraints examples.
Rubric pass rate:
57.9% → 73.4%
The trained 4B model landed at near-parity with Qwen3-235B-A22B-Instruct (73.9%), a model roughly 60x its size.
Learn more:
surgehq.ai/blog/complexconst…
4
30
2,453
echen retweeted
Introducing DAYJOB: our latest benchmark suite for economically valuable agents.
Models can already do impressive work when the task is clear: summarize a document, clean up writing, run a calculation, build a slide.
The next milestone is end-to-end professional judgment: figuring out what needs to be done, making informed decisions, and reasoning through the work over days. That’s what DAYJOB measures.
GDPval was an important step toward measuring whether AI can do economically valuable professional work. DAYJOB pushes that idea further with shorter prompts, messier environments and longer horizons.
Compared with GDPval, DAYJOB changes the shape of the work:
→ Prompts shrink from 337 words to 49–81 words
→ Working environments grow from 1.2 files to 20–26 files
→ Estimated human work grows from 4 hours to 14–17 hours
On DAYJOB, frontier agents still have a long way to go. The strongest system completes only 24.7% of Healthcare assignments and 23.9% of Finance assignments.
Today we’re launching DAYJOB: Healthcare and DAYJOB: Finance, with more domains to follow.
3
2
41
2,645
can agents survive a 9 to 5? introducing DAYJOB, our new benchmark for economically valuable agents.
GDPval asks: “Please reformat this document. Here are 337 words explaining where to look, what analysis to run, and exactly what the finished thing should look like.”
real life: “@bob is that ready yet"
("that" = a forecast spread across 10 spreadsheets, 8 docs, a slack thread where finance is fighting with ds, and a file called model_FINAL_v3_DONT_USE_THIS.xlsx)
DAYJOB is our new benchmark family for knowledge work agents: realistic requests, messy environments, and end-to-end professional judgment.
instead of being told exactly how to do the task, we tell the model what Bob gets told - and reasoning through the rest is the test.
~~
after all, models can do extraordinary things when the task is clear.
summarize a document. clean up an email.
solve navier-stokes!
but somehow we have not yet automated Bob.
the real world is rarely clear. steps are easy; what's hard is the messy end-to-end judgment around them.
real work doesn't arrive with step-by-step instructions on what to do. you get a short vague request. then you have to:
→ figure out what "that" means
→ find which files matter
→ know when to ignore what sally in finance wrote in slack
→ decide which assumptions are outdated
→ notice on day 3 that something from day 1 was wrong
GDPval was a big deal. it was an important step toward measuring real professional work.
but its prompts average 337 words and mostly tell the model how to do the job. the typical task comes with one file. in openai's own experiments, when they shortened the prompts, scores dropped because models struggled to "figure out context."
figuring out context is most of bob's day.
so we built DAYJOB. prompts are short (5x shorter!) and realistic. it comes in much larger environments, averaging 20 to 26 files, and take 20+ hours of expert work. the model has to figure out the job, not just execute a checklist.
~~
today, we’re launching with DAYJOB: Healthcare + DAYJOB: Finance, with more professions coming soon.
one task: review a fund's reports before they go to a new client and flag anything that should stop them.
buried in the pack is a position priced off a bloomberg screenshot. the price was in south african cents. someone entered it as rand.
$260k became $26m.
we collected 66 trajectories across 22 model configs. only 2 caught it.
congrats opus.
everyone else did a lot of impressive work around a $26m mistake.
~~
so can agents survive a 9 to 5?
the best model passes 24.7% of healthcare assignments and 23.9% of finance.
bob's job is harder than navier-stokes.
11
9
94
6,642
Read the blog post: surgehq.ai/blog/dayjob
DAYJOB: Healthcare: surgehq.ai/benchmarks/dayjob…
DAYJOB: Finance: surgehq.ai/benchmarks/dayjob…
1
2
429
one of my favorite examples from our Kimi coding post-training run:
it had to write a Zstandard decompressor, but there was no zstd binary available to check whether it worked.
so the trained model wrote a compressor first, generated its own valid test files, and used those to test the decompressor.
no ground truth existed, so it built one itself.
Read the full report here:
surgehq.ai/blog/hill-climbin…
3
3
42
2,379
echen retweeted
Anthropic recently published new work on automated alignment researchers: Claude agents that search the literature, propose alignment methods, train models, evaluate the results, and iterate.
Across ten alignment failures—including deception, sycophancy, jailbreaks, privacy violations, and reward hacking—the automated researchers found methods that improved safety benchmarks while preserving general capabilities. The strongest methods also generalized to held-out benchmarks, open-ended Petri audits, and models up to 4.7× larger than the models they optimized against.
We contributed to Anthropic's research by building and running the human researcher baseline.
Anthropic compared its automated researchers with ideas from 28 experienced technical AI safety researchers, each given up to eight hours to propose a method for addressing the same alignment failures. Surge ran that pipeline end to end, including researcher recruitment, structured submissions, quality control, and expert review. As the paper puts it: “The human baseline is collected with Surge AI, whose pipeline the study runs through end to end.”
The automated researchers ultimately found methods that outperformed the human-proposed baselines on the seven alignment failures where human ideas were collected. Anthropic is careful about the comparison—the agents could iterate repeatedly, while the human researchers submitted one idea—but the work offers a compelling glimpse of how automated research might complement human researchers in the future.
We spend a lot of time at Surge thinking about what happens as models take on increasingly expert work: how to build credible human baselines, how to evaluate work that requires real judgment, and how to turn expert human knowledge into useful training and evaluation signals.
This study continues our research collaboration with Anthropic that goes back to training Claude with expert human feedback, and research on scalable oversight, inverse scaling laws, and Constitutional AI.
We’re glad to have played a small part in this one.
Read Anthropic’s research:
anthropic.com/research/autom…
2
3
18
2,567
i agree with dario.
i used to work on feed algorithms at google, fb, and twitter. we optimized for engagement because it was the only thing we knew how to measure, even when it was only a weak proxy for what people actually wanted.
and then the systems did exactly what we asked! they figured out how to hack engagement in ways nobody intended, and we didn't understand the consequences until later.
for me, that's always been the alignment problem in miniature. that experience is a big part of why i started Surge, and it shaped how i think about data, quality, evals, and the hills we want to climb.
AI makes the same problem even more consequential. we have models that can solve Navier Stokes, that billions of people will talk to and trust, and that can act on their own.
getting them to climb a hill isn't the hart part; they're already incredible at that. the hard part is choosing the right objective - and making sure the data, rewards, and evals actually point at what we want.
so, i agree. if we don't want to repeat the lessons of social media - but with way smarter systems that can act on their own, and convince, at way greater scale - alignment has to be built into the way we train and evaluate models from the beginning.
a lot more work is needed to make sure models stay human-aligned. reach out if you're interested in working on this together.
22
16
188
16,727
An intern knows how to write code. A staff engineer knows how to make it *shippable*.
We post-trained kimi k2.7 and that’s basically what changed.
Before RLing, kimi k2.7 was failing in the last mile: dropping requirements, writing narrow tests, creating regressions
After RL on just 1,700 Surge coding tasks:
+20.0 SWE-Marathon
+14.6 Terminal-Bench 2.1
+12.4 DeepSWE
+10.7 Terminal-Bench 3
+4.7 SWE-Bench Pro
4
3
31
3,755
the data wasn’t designed around these evals.
DeepSWE, SWE-Marathon, and Terminal-Bench 3 didn’t even exist when the tasks were collected.
the trained model also used fewer steps, not more: 150 → 98 on DeepSWE, 102→78 on TB3.
kimi k2.7 already knew how to write code, but post-training made its actual execution less brittle.
Full writeup: surgehq.ai/blog/hill-climbin…
6
546
echen retweeted
Four major frontier models shipped last week: Claude Fable 5.1, Muse Spark 1.3, Gemini 3.8 Flash, and GPT-6 Astra.
We’ve completed Surge benchmark evaluations for the first three; Astra is still running.
Fable 5.1 leads on overall capability. It scores 68.7 on the Tuesday Work Index, our composite measure of frontier AI at work, and takes the top spot on Chartography, HANDBOOK.md, and EnterpriseBench: CoreCraft.
Muse Spark 1.3 tops ComplexConstraints, our benchmark for professional instruction following. Its xHigh operating point reaches 51.9% for $54.25, on the cost-performance Pareto frontier.
Gemini 3.8 Flash makes its biggest move on frontier mathematics. On Riemann-bench, Gemini 3.8 Flash (High) rises from 39.2% to 51.2%, a 12-point generational improvement. At $69.59, it also lands directly on the benchmark’s cost-performance Pareto frontier.
In short:
‣ Fable 5.1 pushes absolute capability higher.
‣ Muse Spark 1.3 combines top-tier complex instruction following with strong efficiency.
‣ Gemini 3.8 Flash delivers a large step forward in mathematical reasoning at competitive cost.
Our Astra evaluation is still running, and we plan to publish results next week.
Read the full analysis: surgehq.ai/blog/fable-5-1-tu…
8
1
43
5,461
echen retweeted
@DeepSeek_AI V4 Pro scores 59.7 on the Tuesday Work Index, our composite measure of frontier AI at work.
That is up 10.6 points from the V4 Pro preview and 12.2 points from the V4 Flash preview.
1
2
8
2,079
echen retweeted
What should AI optimize for?
Not just efficiency. Not just engagement. Not just doing more work for us.
@echen joined @danshipper at @every to talk about building AI that helps humans grow, Surge’s school for AGI, and what remains worth doing ourselves as models become more capable.
2
3
13
3,319
echen retweeted
Taste isn't something you get from facts alone.
All expert work — whether it's designing a website, pushing the boundaries of science, or getting through a Tuesday at the office — depends on judgment: knowing what's good and what to leave out.
We work with experts across hundreds of fields to teach models exactly that.
@FastCompany featured our work in their latest piece on the human side of teaching AI taste.
3
2
22
3,935
taste isn't a reasoning problem. you can't rubric your way from "correct" to "great."
great writing, design, even great science - all of it depends on judgment, values, and knowing what *not* to do.
one of the scary things about AI is that models are going to eventually learn it all. every few months, something i used to think was intrinsically human - creativity, serendipity, insight - models get better at too.
but models have to learn those values from somewhere. their teachers matter a lot.
spoke to @FastCompany about how we think about taste, values, and post-training at @HelloSurgeAI:
fastcompany.com/91591873/the…
2
23
1,314
echen retweeted
SURGE MODEL REPORT: QWEN 3.8 MAX
Qwen 3.8 Max scores 58.7 on our Tuesday Work Index, our composite measure of frontier AI at work. Up +8.6 points from Qwen 3.7 Max and +22.4 points from Qwen 3.5 Plus.
🧵 of performance across our benchmarks:
2
2
12
1,979
echen retweeted
Introducing Tuesday: A Frontier Index for AI at Work
Frontier models can solve extraordinary problems. But can they get through an ordinary workday?
That’s the premise behind Tuesday. We believe that professional intelligence isn’t a single skill, it’s many capabilities layered together. The Tuesday Index measures all these skills together, with a single easy-to-understand score.
Fable 5 — 66.8
GPT-5.6 Sol — 66.7
DeepSeek v4 Pro — 59.7
Qwen 3.8 Max — 58.7
Gemini 3.7 Flash — 58.2
Muse Spark 1.2 — 53.5
A typical Tuesday morning might require reading a chart, following a long policy, tracking constraints across Slack channels, using several tools, and explaining the key business insight to your boss.
→ You need the basics: can models follow instructions, keep context, use tools, and do what you asked?
→ You need the exceptional: can they reason creatively and solve genuinely hard problems?
→ You need broad abilities like long-horizon work, and narrow skills like reading a chart.
→ You need soft skills too. Professional intelligence isn’t just about correctness. It also demands tact, elegance and precision. The things that inspire us need judgment and taste.
Tuesday spans the basics and the exceptional: instruction following, long-context agentic work, professional documents, graphical reasoning, everyday judgment, writing, agents inside realistic companies, and frontier mathematics.
It includes our eight benchmarks today: Chartography, HANDBOOK.md, GDP.pdf, ComplexConstraints, CoreCraft, Hemingway-bench, Antidote, and Riemann-bench. We’ll add more as we build them.
Made with AI
10
8
83
21,424
it's tuesday morning. you have a stack of work in front of you.
read your email. understand the chart from the DS team. tie it back to that Slack thread from last week. send your VP a rec by noon.
which model do you choose?
we built Tuesday: Frontier Work Index to answer that.
1
3
19
1,447
can your model get through Tuesday?
check our frontier work index out:
surgehq.ai/blog/tuesday-fron…
192