Introducing Animation Bench, our first benchmark to evaluate frontier models on web animation reconstruction. Coding agents can already recreate visually plausible web animations, this work evaluates if they can reconstruct behavior of a web page.

AI is getting cheaper more quickly than any other transformative tech in history. At a given level of performance, cost has fallen ~47%/quarter since 2023. That’s 4× faster than DNA sequencing, 6× faster than compute, 18× faster than lithium batteries, and (up to 1973) 54× faster than electricity.

WeirdML v2 is now out! The update includes a bunch of new tasks (now 19 tasks total, up from 6), and results from all the latest models. We now also track api costs and other metadata which give more insight into the different models. The new results are shown in these two figures. The first one shows an overview of the overall results as well as the results on individual tasks, in addition to various metadata. The second figure shows cost vs performance and shows a clear scaling with better results for higher costs. We also have a very varied pareto frontier with 11 models from 6 different companies having the best accuracy for a given cost for at least some of the cost range. Grok 3, Claude Opus 4 and GPT 4.5 are the ones that underperform for their costs, while Gemini pro and o3 pro have the best results at the highest costs. Qwen3 30B3A, grok 3 mini and deepseek R1 also each represent a good chunk of the pareto frontier.

Introducing Benchmark Reviews: our new initiative to audit AI benchmarks. We are launching with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with not enough information for a review.

I made a list of great startups to join. It's called the Breakout List. The list has 92 companies. These are the 20 with 25 or fewer employees: - Hone (@moritz_stephan, @CarloWillem, @oqbrady) - Normal (@ansonyuu, @hudzah) - Standard Intelligence (@G413N, @devanshpandey) - Tacit Labs (@ninklefitz, @AmDroste) - American Terawatt (@atroyn, @rslparker, @aranibatta) - Conduit (@clemvonstengel, @riopopper) - Convergent (Omkar Savant, Vivek Katara, @debnilsur) - Core Automation (@MillionInt, @_arohan_) - Engram (@dan_biderman, @EyubogluSabri, @realJessyLin) - Instinct (@noahrshinn) - Keenable (@styskin, Matthias Petri) - Lumaril (Mark Elliot, Ben Duffield) - Neion Bio (@Dimkell, Sam Levin) - Pangram Labs (@max_spero_, @bradley_emi) - Quadrillion (@echinaceous) - Re (@karnsaroya, @AnandDhillon, @thecliffwhite, @benaneesh) - Ricursive (@annadgoldie, @Azaliamirh) - Sail Research (@neilmovva, @blintzbase) - Trajectory (@rronak_, @michaelelabd, @QuantumArjun) - Watney Robotics (Sean Cheong, Ryan Gannon) Picks from Elad Gil, Charlie Songhurst, Keith Rabois, Mike Vernal, Alana Goyal, Sonya Huang, Ramtin Naimi, Marc Bhargava, Cory Levy, Aashay Sanghvi, Konstantine Buhler, John Luttig, Varun Gupta, Ray Tonsing and Avichal Garg. Disclosure: I'm a small investor in American Terawatt, Convergent, Standard Intelligence and Trajectory (in this post), and in Factory, Physical Intelligence and SF Compute (elsewhere on the list). I didn't vote. The full list is on Breakout List.

We fine-tuned @thinkymachines' Inkling-Small to fix LaTeX compile errors inside the editor, in under a second. On real errors it matches Claude Fable 5.1, six times faster and at about 1/40 of the cost. When we interviewed mathematicians, one of the most common annoyances we heard about in their day to day work is battling with LaTeX compilation errors. The most obvious solution is to ask a chatbot to fix errors. That works decently well, except switching to a different window breaks the writing flow, and copy-pasting the snippet often loses relevant context, especially when multiple files are involved. So we scraped 272,000 questions from TeX.StackExchange and kept the 3,978 whose snippet still fails today, whose accepted answer compiles, and whose difference replays exactly as a patch. The largest dataset previously released had 88. We tried tweaking the reward function in many ways. The first one was only a compilation check, which led the model to remove content until the document built. In the end, what worked best was rewarding compilation and a close match with the solution's PDF, and penalizing removed content. We're rolling out the model inside the @sundialmd editor this week. When a compile fails, the fix is applied as a suggestion in under a second and the PDF rebuilt. Write-up: sundial.md/blog/textinguishe… Thanks @tinkerapi and @thinkymachines for the support.

Today, we publish a Transformative AI Strategy for Europe. Over the last few months, we’ve rallied researchers and engaged with governments to develop a plan for protecting the prosperity, sovereignty, and security of Europeans in a time of rapid AI progress. transformative-ai.eu @MonikaSchnitzer and I are honored to have convened an all-star team of thinkers and researchers contributing ambitious near-term objectives to make Europe relevant again, including: — Creating a Member State Alliance for Supply Chain Security (@anton_d_leicht et al) — Making European institutions ready to act in a transformative AI world (Conor McGlynn et al) — Securing Europe's share of global AI compute, in a ‘European Way’ that benefits local communities (@philip_fox_ et al) — Ensuring resilience to AI crises (@ben_s_bucknall et al) — Making Europe the global leader in assurance technology — And more objectives around security of supply and leverage (@milorignell et al), economic strength (@FraukeStehr et al), and safety/security (@NoemiDreksler et al) Our all-star senior expert council of Europe’s best and brightest (and non-European friends) reviewed drafts, provided strategic advice, and made suggestions for how to make the strategy more useful: @Ph_Aghion @Christophkw @bakkermichiel @ischinger @vestager @aleks_madry @FuestClemens Marta Kwiatkowska @LeoVaradkar @antonosika @DAcemogluMIT @Yoshua_Bengio It’s never been more clear: AI is real, and Europe needs to act. We’ve had all the warning shots and wake-up calls we need. Now the question is ‘what must be done?’ This strategy is our answer. transformative-ai.eu