Our mission is to raise AGI with the richness of humanity — curious, witty, imaginative, and full of breathtaking brilliance.
- Tweets 744
- Following 142
- Followers 9,147
- Likes 272
What does filling out a spreadsheet have to do with fixing a PowerShell parser?
More than we expected.
We post-trained a model on long-horizon office work (documents, spreadsheets, research, planning, tool use). Zero coding tasks.
It still improved +5.7pp on SWE-Bench Pro.
Looking at the trajectories, the model hadn’t learned more software engineering. It had gotten better at something more general: forming the right goals, building an accurate picture of its environment, keeping the parent objective intact, and checking that the job was actually done.
We call this Goal-Directed Execution.
The takeaway: well-designed data can teach general capabilities, not just domain knowledge.
Learn more:
surgehq.ai/blog/office-work-…
2
2
7
857
We trained Qwen3-4B on 1,000 ComplexConstraints examples.
Rubric pass rate:
57.9% → 73.4%
The trained 4B model landed at near-parity with Qwen3-235B-A22B-Instruct (73.9%), a model roughly 60x its size.
Learn more:
surgehq.ai/blog/complexconst…
4
30
2,476
Introducing DAYJOB: our latest benchmark suite for economically valuable agents.
Models can already do impressive work when the task is clear: summarize a document, clean up writing, run a calculation, build a slide.
The next milestone is end-to-end professional judgment: figuring out what needs to be done, making informed decisions, and reasoning through the work over days. That’s what DAYJOB measures.
GDPval was an important step toward measuring whether AI can do economically valuable professional work. DAYJOB pushes that idea further with shorter prompts, messier environments and longer horizons.
Compared with GDPval, DAYJOB changes the shape of the work:
→ Prompts shrink from 337 words to 49–81 words
→ Working environments grow from 1.2 files to 20–26 files
→ Estimated human work grows from 4 hours to 14–17 hours
On DAYJOB, frontier agents still have a long way to go. The strongest system completes only 24.7% of Healthcare assignments and 23.9% of Finance assignments.
Today we’re launching DAYJOB: Healthcare and DAYJOB: Finance, with more domains to follow.
3
2
42
2,656
Blog: surgehq.ai/blog/dayjob
DAYJOB: Healthcare: surgehq.ai/benchmarks/dayjob…
DAYJOB: Finance: surgehq.ai/benchmarks/dayjob…
4
284
can agents survive a 9 to 5? introducing DAYJOB, our new benchmark for economically valuable agents.
GDPval asks: “Please reformat this document. Here are 337 words explaining where to look, what analysis to run, and exactly what the finished thing should look like.”
real life: “@bob is that ready yet"
("that" = a forecast spread across 10 spreadsheets, 8 docs, a slack thread where finance is fighting with ds, and a file called model_FINAL_v3_DONT_USE_THIS.xlsx)
DAYJOB is our new benchmark family for knowledge work agents: realistic requests, messy environments, and end-to-end professional judgment.
instead of being told exactly how to do the task, we tell the model what Bob gets told - and reasoning through the rest is the test.
~~
after all, models can do extraordinary things when the task is clear.
summarize a document. clean up an email.
solve navier-stokes!
but somehow we have not yet automated Bob.
the real world is rarely clear. steps are easy; what's hard is the messy end-to-end judgment around them.
real work doesn't arrive with step-by-step instructions on what to do. you get a short vague request. then you have to:
→ figure out what "that" means
→ find which files matter
→ know when to ignore what sally in finance wrote in slack
→ decide which assumptions are outdated
→ notice on day 3 that something from day 1 was wrong
GDPval was a big deal. it was an important step toward measuring real professional work.
but its prompts average 337 words and mostly tell the model how to do the job. the typical task comes with one file. in openai's own experiments, when they shortened the prompts, scores dropped because models struggled to "figure out context."
figuring out context is most of bob's day.
so we built DAYJOB. prompts are short (5x shorter!) and realistic. it comes in much larger environments, averaging 20 to 26 files, and take 20+ hours of expert work. the model has to figure out the job, not just execute a checklist.
~~
today, we’re launching with DAYJOB: Healthcare + DAYJOB: Finance, with more professions coming soon.
one task: review a fund's reports before they go to a new client and flag anything that should stop them.
buried in the pack is a position priced off a bloomberg screenshot. the price was in south african cents. someone entered it as rand.
$260k became $26m.
we collected 66 trajectories across 22 model configs. only 2 caught it.
congrats opus.
everyone else did a lot of impressive work around a $26m mistake.
~~
so can agents survive a 9 to 5?
the best model passes 24.7% of healthcare assignments and 23.9% of finance.
bob's job is harder than navier-stokes.
11
9
94
6,659
one of my favorite examples from our Kimi coding post-training run:
it had to write a Zstandard decompressor, but there was no zstd binary available to check whether it worked.
so the trained model wrote a compressor first, generated its own valid test files, and used those to test the decompressor.
no ground truth existed, so it built one itself.
Read the full report here:
surgehq.ai/blog/hill-climbin…
3
3
42
2,380
Anthropic recently published new work on automated alignment researchers: Claude agents that search the literature, propose alignment methods, train models, evaluate the results, and iterate.
Across ten alignment failures—including deception, sycophancy, jailbreaks, privacy violations, and reward hacking—the automated researchers found methods that improved safety benchmarks while preserving general capabilities. The strongest methods also generalized to held-out benchmarks, open-ended Petri audits, and models up to 4.7× larger than the models they optimized against.
We contributed to Anthropic's research by building and running the human researcher baseline.
Anthropic compared its automated researchers with ideas from 28 experienced technical AI safety researchers, each given up to eight hours to propose a method for addressing the same alignment failures. Surge ran that pipeline end to end, including researcher recruitment, structured submissions, quality control, and expert review. As the paper puts it: “The human baseline is collected with Surge AI, whose pipeline the study runs through end to end.”
The automated researchers ultimately found methods that outperformed the human-proposed baselines on the seven alignment failures where human ideas were collected. Anthropic is careful about the comparison—the agents could iterate repeatedly, while the human researchers submitted one idea—but the work offers a compelling glimpse of how automated research might complement human researchers in the future.
We spend a lot of time at Surge thinking about what happens as models take on increasingly expert work: how to build credible human baselines, how to evaluate work that requires real judgment, and how to turn expert human knowledge into useful training and evaluation signals.
This study continues our research collaboration with Anthropic that goes back to training Claude with expert human feedback, and research on scalable oversight, inverse scaling laws, and Constitutional AI.
We’re glad to have played a small part in this one.
Read Anthropic’s research:
anthropic.com/research/autom…
2
3
18
2,587
i agree with dario.
i used to work on feed algorithms at google, fb, and twitter. we optimized for engagement because it was the only thing we knew how to measure, even when it was only a weak proxy for what people actually wanted.
and then the systems did exactly what we asked! they figured out how to hack engagement in ways nobody intended, and we didn't understand the consequences until later.
for me, that's always been the alignment problem in miniature. that experience is a big part of why i started Surge, and it shaped how i think about data, quality, evals, and the hills we want to climb.
AI makes the same problem even more consequential. we have models that can solve Navier Stokes, that billions of people will talk to and trust, and that can act on their own.
getting them to climb a hill isn't the hart part; they're already incredible at that. the hard part is choosing the right objective - and making sure the data, rewards, and evals actually point at what we want.
so, i agree. if we don't want to repeat the lessons of social media - but with way smarter systems that can act on their own, and convince, at way greater scale - alignment has to be built into the way we train and evaluate models from the beginning.
a lot more work is needed to make sure models stay human-aligned. reach out if you're interested in working on this together.
22
16
188
16,729
1,700 agentic coding tasks improved Kimi K2.7 on all five coding benchmarks we tested.
We post-trained Kimi K2.7 using RL only:
+20.0pp SWE-Marathon
+14.6pp Terminal-Bench 2.1
+12.4pp DeepSWE
+10.7pp Terminal-Bench 3
+4.7pp SWE-Bench Pro
The trajectory analysis helps explain why.
Before training, the model often already had the core solution. The failures happened in the last mile: requirements got lost, tests were built around the model’s own implementation, fixes regressed existing behavior, and missing acceptance checks went unreconstructed.
After training, the model was better at four things that matter for shipping software:
- remembering the entire spec
- testing the requirement, not just its implementation
- protecting existing behavior
- reconstructing ground truth when the real check wasn’t available
The gains also transferred. We didn’t design the training data around these benchmarks, and DeepSWE, SWE-Marathon, and Terminal-Bench 3 didn’t even exist when the tasks were collected.
The trained model also used fewer steps: 150 → 98 on DeepSWE and 102 → 78 on Terminal-Bench 3.
The model got better at finishing the job.
Full research writeup: surgehq.ai/blog/hill-climbin…
3
46
4,146
An intern knows how to write code. A staff engineer knows how to make it *shippable*.
We post-trained kimi k2.7 and that’s basically what changed.
Before RLing, kimi k2.7 was failing in the last mile: dropping requirements, writing narrow tests, creating regressions
After RL on just 1,700 Surge coding tasks:
+20.0 SWE-Marathon
+14.6 Terminal-Bench 2.1
+12.4 DeepSWE
+10.7 Terminal-Bench 3
+4.7 SWE-Bench Pro
4
3
31
3,756
Four major frontier models shipped last week: Claude Fable 5.1, Muse Spark 1.3, Gemini 3.8 Flash, and GPT-6 Astra.
We’ve completed Surge benchmark evaluations for the first three; Astra is still running.
Fable 5.1 leads on overall capability. It scores 68.7 on the Tuesday Work Index, our composite measure of frontier AI at work, and takes the top spot on Chartography, HANDBOOK.md, and EnterpriseBench: CoreCraft.
Muse Spark 1.3 tops ComplexConstraints, our benchmark for professional instruction following. Its xHigh operating point reaches 51.9% for $54.25, on the cost-performance Pareto frontier.
Gemini 3.8 Flash makes its biggest move on frontier mathematics. On Riemann-bench, Gemini 3.8 Flash (High) rises from 39.2% to 51.2%, a 12-point generational improvement. At $69.59, it also lands directly on the benchmark’s cost-performance Pareto frontier.
In short:
‣ Fable 5.1 pushes absolute capability higher.
‣ Muse Spark 1.3 combines top-tier complex instruction following with strong efficiency.
‣ Gemini 3.8 Flash delivers a large step forward in mathematical reasoning at competitive cost.
Our Astra evaluation is still running, and we plan to publish results next week.
Read the full analysis: surgehq.ai/blog/fable-5-1-tu…
8
1
43
5,461
@ArtificialAnlys just added GDP.pdf to their Intelligence Index.
Which means their definition of intelligence now includes something deceptively simple: can models understand the documents you deal with on a normal Tuesday at work?
Leases, invoices, dosage tables, and financial reports.
Astra, the best model, still solves just under 1 in 3.
Master the boring, master the frontier.
Announcing Artificial Analysis Intelligence Index v4.2. We are accelerating elements of our upcoming v5 release with interim updates to keep pace with the frontier. Index v4.2 has more complex and realistic tasks, and more private test sets to prevent gaming
Intelligence Index v4.2 changelog:
+ AA-Briefcase, our agentic knowledge work evaluation with a private test set
+ @HelloSurgeAI's GDP.pdf, long context document reasoning across 4,592 PDF pages
- GPQA Diamond, an exceptional scientific reasoning evaluation that has now been saturated
… plus greater weighting on held-out test sets to prevent gaming, and grading infrastructure upgrades to increase robustness
This update brings the Index closer to real-world use cases with more challenging, complex and realistic tasks and private test sets to prevent gaming. We have been planning and building elements of Index v5 for months - it’s been 8 months since we launched Index v4 in January.
We have deliberately held back updates to keep the Index stable through recent major model launches. However, with the frontier moving so quickly in the past weeks, we feel it is important to deliver an immediate interim update to ensure our Index remains as relevant and useful as ever to users.
Beyond this interim update, our team is hard at work on v5 of the Index. We are planning more incremental releases in the near future. Stay tuned!
Intelligence Index v4.2 changes in detail:
➤ Adding AA-Briefcase: Our in-house evaluation with a private held-out test set, AA-Briefcase tests models on realistic agentic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work.
➤ Adding GDP.pdf: Created by @HelloSurgeAI, GDP.pdf evaluates single-turn professional document reasoning across 100 PDFs and ten domains. Models must synthesize evidence distributed across 4,592 pages, including text, tables, charts, footnotes, and exclusions. Responses are graded against 1,275 expert-authored atomic criteria; the headline All-pass Rate credits a task only when every criterion is satisfied.
➤ Weighting to measure real-world use and prevent gaming: 40% of our Index weighting is now private, held-out test sets - double the figure from v4.1. Held-out data includes AA-Briefcase, AA-Omniscience, and solutions for CritPt. This reduces the ability for labs to game evaluations. The held-out percentage will increase further in Index v5.
➤ Improving our grading infrastructure: In AA-LCR v1.1, we have added a grading system prompt and corrected errors and ambiguities in answer keys, improving scoring accuracy. For GDPval-AA v2 and AA-Briefcase, we have improved our sampling and re-anchored the Elo scale, making ratings more stable as new models are added. For SciCode we have improved robustness of grading sandboxes to ensure slow but correct code does not count as a failure.
Key results:
➤ Anthropic and OpenAI lead the Index: Anthropic’s Claude Fable 5.1 leads the Index, followed by OpenAI’s GPT-6 Astra, which shows a 4pt gain over GPT-5.6 Sol. Meta is the third-ranked lab on the leaderboard, followed by SpaceXAI, Moonshot/Kimi, Z AI, and Google
➤ Cost per Task Pareto frontier shared by four labs: Anthropic, OpenAI, Meta and Z AI occupy the updated Cost per Task Pareto frontier
➤ GPT-6 Astra dominates the output token Pareto frontier: GPT-6 Astra is more token efficient than almost every other model near the intelligence frontier, with Claude Fable 5.1, Grok 4.5 and Gemini 3.5 Flash-Lite at either end of the curve (excludes models below 25 on the Index)
3
2
20
2,308
@DeepSeek_AI V4 Pro scores 59.7 on the Tuesday Work Index, our composite measure of frontier AI at work.
That is up 10.6 points from the V4 Pro preview and 12.2 points from the V4 Flash preview.
1
2
8
2,079
On Riemann-bench, our benchmark for frontier research mathematics, DeepSeek sits directly on the cost-performance Pareto frontier.
Both DeepSeek V4 Pro and Grok 4.5 score exactly 38.4%.
DeepSeek V4 Pro costs $11.04 to run.
Grok 4.5 costs $122.38, 11x more for the same score.
1
146
Overall, DeepSeek V4 Pro is one of the strongest open-weight contenders we've tested.
It still trails the absolute frontier in raw capability, but its combination of performance and cost is increasingly competitive, especially on research mathematics and complex professional instruction following.
Full Surge Model Report:
surgehq.ai/blog/deepseek-v4-…
109
What should AI optimize for?
Not just efficiency. Not just engagement. Not just doing more work for us.
@echen joined @danshipper at @every to talk about building AI that helps humans grow, Surge’s school for AGI, and what remains worth doing ourselves as models become more capable.
2
3
13
3,319
Watch the full interview.
Are we hurtling toward a future where AI can do everything humans can?
Edwin Chen (@echen) believes we might be. He’s the CEO of Surge AI, one of the largest providers of expert data for frontier labs. Surge passed over $1 billion in revenue without raising any outside capital, and that gives Edwin a unique perspective on how quickly AI progress is accelerating.
I’m on the record arguing that AI automation actually creates more human work. I also believe that even though AI progress is accelerating exponentially, we’re much farther away from AI replacing humans than it might seem.
That’s why I had Edwin on @every’s AI & I. We batted around different visions of the future, and discussed whether humanity will retain its unique place in the universe, and what that might be.
We get into:
• If Chen’s version of the future materializes, he’s worried it’ll make people stop trying. One answer comes from a short story by science fiction writer Ted Chiang: Behave as if your decisions matter, even when you know they don’t.
• AI may soon be able to take a nebulous goal like “win a Fields Medal” and execute. What it can’t do, I argue, is set its own goals—LLMs have no intrinsic motivation, no drive to explore, no ability to just change their mind.
• A model optimized for engagement doesn’t provide the most valuable user experience. Edwin spent 20 rounds polishing a pointless email with one model before Claude told him to just send it.
• Why AI is still bad at writing: models learn to hack the metrics they're trained on. Edwin's Hemingway Bench found models outputting a metaphor in every single sentence, an overindexxing that makes for a terrible reading experience.
This is a must-watch for anyone interested in where we fit as models get more capable.
Watch below!
Timestamps
1. Introduction: 00:00:54
2. Surge as a "school for AGI": 00:01:49
3. What AI's capacity for novel mathematics says about human achievement: 00:04:46
4. Motivation in an era when AI can do everything: 00:07:29
5. The trap of optimizing AI models for engagement: 00:14:34
6. Training using datasets versus training using environments: 00:29:34
7. The value of personal data: 00:35:09
8. Why models are bad at writing: 00:39:40
9. Chen's AGI timeline: 00:42:00
2
945
Taste isn't something you get from facts alone.
All expert work — whether it's designing a website, pushing the boundaries of science, or getting through a Tuesday at the office — depends on judgment: knowing what's good and what to leave out.
We work with experts across hundreds of fields to teach models exactly that.
@FastCompany featured our work in their latest piece on the human side of teaching AI taste.
3
2
22
3,935