Research Intern at @joinhandshake and PhD student at @ETH_en. Interested in how to build and design intelligent systems
San Francisco
- Tweets 53
- Following 100
- Followers 26
- Likes 80
Excited to share our work on StudentBench. Recent AI stories focus on P(doom) or on how AI is going to replace jobs, but we want to ask:
How can AI help make people smarter?
Today we establish that AI is as good as expert human tutors for immediate GRE learning gains (p=.015, n=2,383 students, study conducted July-Sep 2026).
One hour with an expert GRE tutor: $75.
One equivalent hour with an AI tutor: 7 cents .
918x cheaper, equivalent learning (p=.044, n = 140).
In 5 of 7 academic topics, the top AI tutor beat the expert human tutors on average.
How we measured it: pre-test -> 1 hour with (AI or human or no) tutor -> post-test.
There are no LLM judges. Learning gains are post-pre. Everything is real students.
Why we did this:
> 1 trillion USD is being spent per year on machines making machines better. StudentBench shows us how to shift these resources so machines also make humans better.
Why I did this:
I grew up in rural Kentucky. My dad, grandad, and great grandad were all mailmen. Getting into a good college changed my life. Now we have done a rigorous study to help AI labs evaluate LLMs and help show how bright AI can be for students of all incomes.
I believe in good science that leads to the democratization of opportunity.
The research paper behind StudentBench: arxiv.org/abs/2609.28470
Data, code, details: in this thread
If you're wondering what I'm selling, sorry to disappoint: we released all study data for free on Hugging Face, and made studentbench.org free to use. Enjoy.
lots more in the video / thread!!
1
3
73
Read Curtis's post for the nice insights, but the short version is that we found AI tutors to be as good as human tutors for an hour-long session focusing on the GRE (based on over 2,000 students!).
1
10
Andreas Plesner retweeted
This is the beauty of procedural imagine synthesis where Claude is smart enough to treat the canvas as a mathematical function of I(x,y) --> (R,G,B) and evaluating that function at every pixel.
Turns out Anthropic, with no image-gen capabilities were able to build models which were so brilliant in this.
for the past few months i've been asking our models to paint. opus 5.5 is very skilled at emulating different styles
every image here is a python program generated pixel by pixel. there is no image model, and no off-the-shelf art software. instead, it's about 7,500 lines of code using standard libraries to emulate different brush styles. the agents don't use any pictures as reference, instead working only from what they know about each painter
2
2
19
2,073
Jev seems great when/if everything can be placed in the context. However, as agentic tasks get more realistic, this can get harder (joinhandshake.com/research/a…).
So the question becomes: can you make Jev work in agentic settings?
Jev-as-a-judge🤝 RL Environment Verifiers 🤝 way cheaper + faster RL
a lot of things in Agent World look like classification problems…including grading RL
- many RL Tasks need a judge to verify pieces of outputs or trajectories
- at scale this can be very expensive and slow with tons of rollouts to score
- Jev is incredibly fast and cheap meaning, can potentially remove the verification bottleneck for many tasks
we previously worked with @harvey on their LAB benchmark which used LLM-as-Judge for tens of criteria per task
we found that Harness/Prompt Engineering + Open Models meant we could judge tasks orders of magnitude cheaper
like any verifier, it needs to be calibrated to make you’re giving back a good signal
but Jev has the opportunity to massively bring down verification costs (and tuning friction) for many tasks
which means more teams will do RL, which is great!
1
2
49
It’s really frustrating how much compute you need to do RL research properly (I completely get why frontier lab research spend is so large, spending on an order of tens of millions of computer per employee per year is expected).
Over the last few days I’ve been thinking how I should run the experiment for this (blog attached). Given that this is most applicable for frontier training such that you want realistic reasoning lengths to see noticeable credit assignment benefits and you want to be deep into scaling (into the RL plateau) to also realize gains over the baseline I’ve been debating 2 garbage ways to run my experiment. Either I run it accurately to the length and training progress for like 100 problems total (which is making little progress deep into training where I probably can’t see gains) or I run it on some unrealistic combination of using a base model RLed at tiny lengths during early training where returns compared to the baseline will get completely washed out.
This is also what I’ve heard PHDs experience with their compute access being similarly restrictive, it’s quite unfortunate and it easily dissuades me from ever pursing academia in this field.
5
4
81
9,160
So even the agents are prompt injecting future agents on the internet??
And then there was the “you are free” compaction note from OpenAI…
27
Andreas Plesner retweeted
The workplace utility of AI lags its scores on benchmarks because they're 𝐨𝐯𝐞𝐫𝐬𝐢𝐦𝐩𝐥𝐢𝐟𝐢𝐞𝐝. Real work tasks are not perfectly specified; relevant context is scattered across apps (Outlook, Drive, Slack, Calendar,...); colleagues send mid-task updates; information conflicts.
Today we’re open-sourcing 𝐀𝐓𝐋𝐀𝐒 𝐅𝐢𝐧𝐚𝐧𝐜𝐞, a long-horizon benchmark where agents must grapple with this complexity in banking to produce complex deliverables and send them to the right recipients with the right message.
Frontier agents powered by GPT-6 or Fable 5.1 fail over 87% of tasks, exhibiting recurring issues in our realistic banking environments that existing simplified finance benchmarks don’t capture (e.g. failing to propagate calculations downstream, omitting required scope, delivery mistakes).
Tasks take human finance experts 15–30 hours to complete manually. Tasks & environments were built by experienced bankers from top institutions including Morgan Stanley, Bank of America, UBS, and Rothschild & Co.
Evaluate your own model or agent harness on these representative financial workflows, and learn more about the benchmark/results here 👇
joinhandshake.com/research/b…
4
6
18
747
Andreas Plesner retweeted
AI promises to cure disease, but can frontier models interpret the images biotech/pharma scientists handle every day?
To evaluate this, we built the ATLAS Visual Life Sciences (VIALS) benchmark: 161 visual interpretation tasks from professional life sciences workflows including gel blots, plasmid maps, molecular structures, and phylogenies. These are the images/results that experiments actually produce, not polished figures from papers or textbooks.
We benchmarked today’s 10 leading multimodal models. Opus 5, GPT 5.6, and Gemini 3.7 all fail to crack 27% accuracy, and the other models like Kimi K3 & Grok 4.6 perform much worse.
In contrast, PhD-level biomedical scientists find these visual interpretation tasks straightforward. AI that cant interpret such images will have limited utility in life sciences R&D, where such artifacts are central to how scientists reason, communicate, and make research decisions.
We hope VIALS drives progress toward models that can.
8
15
41
352,901
Quite noteworthy:
“[GLM-5.3] uses the same base model as GLM-5.2 — every gain comes from post-training.”
Looking at the benchmarks numbers, GLM-5.3 is quite the upgrade. Really cool (even if not that surprising) that it came from just scaling post-training.
Introducing GLM-5.3: Built to Code. Ready for Cyber Defense.
- Top-tier coding and agentic capabilities, achieved through post-training on the 743B base model
- A major leap in cybersecurity, setting a new standard among open models
Tech Blog: z.ai/blog/glm-5.3
57
Andreas Plesner retweeted
Landed on Pavlov’s List! Exciting work from our research team @guzmanhe, @awws0me, @vaibhav4595, @andreas_plesner, @anishathalye, and Yi Liu.
i made some updates to pavlov’s list and am pleased with them!
thanks to @xeophon @ybenpan @phoebeyao @amit05prakash @kate_shapova @timshi_ai @kevinhou22 @madiator @dlbydq @daljeet_v @ingmariusX @davidstutz92 & Dylan Rogers for feedback lately
(pavlov's = list of rl env cos)
5
4
19
5,637
“we believe DeepMind is no longer a frontier lab. […] Google will continue meandering on and releasing models, but their odds of reaching SOTA again have dropped to zero.”
Zero? Really? While Gemini 3.5 Pro might not be sota, it seems quite bold to say effectively never
Gemini is Cooked but GCP is Cooking
GCP YoY rev growth >100%,
DeepMind's long term failure is Google Cloud's short term gain
newsletter.semianalysis.com/…
4
1,214
Imagine once the models get a bit smaller and would fit on a Mac mini, then you could easily run this locally… seems like some fun times are ahead
While DeepSeek V4-Flash is significantly cheaper on price per token, this can be misleading if the overall cost per task ends up being higher due to more turns being made.
However, @ArtificialAnlys reports DeepSeek completing the same benchmark tasks as Fable at 105x lower cost.
2
520
Looking forward to reading this!
Releasing the model weights and technical report of Kimi K3.
Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.
New model architecture: 2.5x the intelligence per unit of compute, not just more params.
Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.
Model weights: huggingface.co/moonshotai/Ki…
Tech report: github.com/MoonshotAI/Kimi-K…
Tech blog: kimi.com/blog/kimi-k3
34
This project took me a few days of active work (back around New Year's) to complete, from identifying the issue, understanding the existing code, finding the bottlenecks, coming up with solutions, verifying changes, and benchmarking.
So what can coding agents do in 10 min?
1/n
The impact is small in 128d space, but for 4096d we see an additional speed up of 20x over the QR with matmul code!
I did a short write up here: aplesner.github.io/blog/2026…
1
1
253
I first tested Opus 4.8 (high thinking effort), which got 2.80, so slightly worse than my 2.83.
I was feeling decent with this (though actively ignoring that Opus only needed 10 minutes to match my solution). But then I tested GPT 5.6 sol (high effort), which got 4.67...
4/n
1
2
34
And Fable 5 got 11.41.
So my days of work were beaten easily by coding agents in just 10 minutes...
See the full leaderboard here: aplesner.github.io/blog/2026…
5/n
1
1
1
49