25Y in startups/VC. CEO AutoBench, eZecute, SBC FoodTech, Agrifood-tech Italia. M.Sc. Physics. 1/2 Polish, 1/2 S. African, born in Milan, grownup Roman.
Rome, Italy
- Tweets 15,238
- Following 484
- Followers 14,560
- Likes 3,125
Pinned Tweet
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck.
Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic.
Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions.
Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick.
Luna is already sitting in it. Every router we tested is outside it.
It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model.
1/14 👇
3
1
13
1,829
On Friday I promised some good news for @AnthropicAI and @OpenAI.
Monday is in and the news is not Anthropic’s SEC filing. The real good news is that LLM routers work really badly (check our benchmark below).
Frontier LLM revenues are safe. For now…
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck.
Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic.
Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions.
Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick.
Luna is already sitting in it. Every router we tested is outside it.
It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model.
1/14 👇
1
4
810
I have some good news and bad news for @AnthropicAI and @OpenAI.
Let's start with the good news (on Monday...)
10
7
149
140,507
The good news…
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck.
Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic.
Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions.
Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick.
Luna is already sitting in it. Every router we tested is outside it.
It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model.
1/14 👇
1
144
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck.
Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic.
Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions.
Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick.
Luna is already sitting in it. Every router we tested is outside it.
It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model.
1/14 👇
3
1
13
1,829
That is the story on the board. The prize is real. Nobody is collecting it.
That is the good news for frontier-model businesses.
And, yet, there is no good news without bad news.
More on Thursday.
13/14
1
1
112
Who are we? AutoBench is the AI evaluation infrastructure for the agentic era.
We use multi-LLM evaluation (Collective-LLM-as-a-Judge) to measure quality, cost, and speed on live, open-ended prompts (the kind agents actually send).
Interested in what we do?
Reach out here, or on autobench.org, or on our HF community: huggingface.co/AutoBench
14/14 end🧵
2
121
<< Per alimentare l'ostilità verso i rifugiati ucraini in Italia, la squadra ha proposto di diffondere una fake su come "le ville sequestrate appartenenti ai russi saranno consegnate ai rifugiati ucraini, con il costo della loro manutenzione proveniente dai bilanci municipali". Inoltre, i blogger italiani dovevano essere pagati per produrre post sul danno causato dalle sanzioni contro la Russia. "La cosa principale è che non capiscono chi l'ha commissionato - in questo momento, pensano che sia un partito politico italiano", ha avvertito un dipendente dell'ASD.>>
È raro ma, ogni tanto, vedrai dei giornalisti dimostrare che la categoria ha ancora una funzione (e una dignità). E questa indagine di The Insider sull’organizzazione moscovita che confeziona e propala le fake russe (parola che usano esplicitamente) è da Pulitzer: nomi, documenti, episodi, ecc.
La domanda, a questo punto, è: chi sono i blogger italiani di cui sopra?
Read about how @JDVance and others fell for a lie about Volodymyr Zelensky concocted by agents of Russia's Presidential Administration. And how the Vice President's falling for it was collected in a file sent directly to the Kremlin. Inside Putin's "Doppelgänger" operation, based on leaked correspondence between the PA and Russia's Agency for Social Design: theins.press/en/inv/298359
8
145
433
9,787
France is financially screwed, but it still maintains a massive security asset (nuclear deterrence). Whoever wins the next presidential elections will be handed the same deal: extend France’s security and deterrence apparatus to the rest of Europe in exchange for a feasible path to financial sustainability. This is something even Marine Le Pen understands well (not the Melenchonian left - totally beyond their mental faculties)
14
24
338
38,450