We audited 15 benchmarks and labeled 9 flawed: - In Terminal Bench 4.0 we found 45.5% of tasks to be broken after reviewing github issues. - In HLE, 46% of the 48 questions we randomly sampled were broken. - In DeepSWE 1.1 we found a bug that can break grading for every task.

New paper: How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks Frontier models have tackled Navier–Stokes. People naturally wonder, how is their capability for physical sciences? According to today’s benchmarks, frontier models still struggle with physics (e.g. 47.3% with GPT5.6-sol on Humanity Last Exam - Physics component). But that picture did not match physicists' experience using these models. So we did a rigorous analysis of current benchmarks and re-graded their supposed failures with Yale physicists. We found that the problem was often the benchmark and not the model !! When corrected, the models almost saturate all benchmarks, including everyone’s favourite, Humanity Last Exam (physics-part) and Critpt. These failures in turn make AA less reliable as a measure of model’s true capability. For more details, results and analysis check out our preprint: arxiv.org/abs/2609.13009 and blog post: jsous.github.io/blogs/is-phy…

There's been some recent suggestions that recent AIxMath successes have relied on stealing ideas from mathematicians' work in progress. While we can't know for sure what happened, I think the evidence for this is very weak. It seems to me that the primary function these claims have is to deny or downplay recent growth in model capabilities. Whatever one thinks of the labs, this is exceedingly dangerous to the health of profession; if human mathematics is going to survive we have to face facts about what the models can do. And we should also celebrate exciting mathematics, even as we're concerned about mathematicians and mathematical expertise; who are these results for, if not us? The weaker claim that PDFs or announcements put out by labs and lab employees often do not adequately attribute or credit other work seems correct to me. How important is it? I wish that science was free from concerns of human ego but it's not. (Maybe it soon will be.) Top scientists mostly forgo substantial monetary compensation for their work, and are instead primarily compensated in credit. That some people are motivated by this does not make their work less important, and credit and prestige have historically incentivized good work. Of course I wish that we were all perfect truth-seekers and cared about this less. Will credit continue to matter so much as model capabilities increase? I'm not sure; I very much hope not. I think this is more a question about the shape of society than the shape of model capabilities. That there is so much jockeying for credit between the labs, and people at the labs, suggests to me that the people closest to the technology might think it still matters. On a related note, I find the race dynamics between the labs disturbing. While I know many people at the labs care about science and the norms of good science, it seems to me that these norms were sacrificed as soon as there was a hint that there was a (frankly pretty inconsequential, though mathematically cool) prize to be won. It makes me worry how they will behave if something more substantial is at stake, and what principles will be sacrificed for that.

Astra made me a sonar app that emits undetectable audio to scroll up/down on your computer. It uses the doppler effect to determine where your hand placement is. You can even double tap in the air to change scroll directions!

Introducing Kimi K3: Open Frontier Intelligence 🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal 🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts 🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost 🔹 Built for long-horizon agentic coding and self-evolving workflows Kimi K3 is now live on on Kimi.com, Kimi Work, Kimi Code, and the Kimi API. Open Weights by July 27, 2026. 🔗 API: platform.kimi.ai 🔗 Tech blog: kimi.com/blog/kimi-k3

Fable 5 thinks we are idiots

We’ve received notice that the Department of Commerce has lifted export controls on Claude Fable 5 and Mythos 5. We'll begin restoring access tomorrow, and will share an update soon. We’re grateful to our users for their patience, and to everyone who worked with us on redeploying the models.

Introducing two new research-level mathematical training datasets! Training data for research mathematics, especially in the post-training regime, is severely lacking. Using our benchmark pipelines on a larger scale, we now created almost 6,000 training data samples.

Today, MIT & the IMO released MathNet, the world’s largest dataset of International Math Olympiad problems & solutions 🌍 MathNet is 5x larger than previous datasets & is sourced from over 40 countries across 4 decades: bit.ly/4u1bhBC

PRBench: End-to-end Paper Reproduction in Physics Research - Presents a benchmark of 30 expert-curated tasks spanning 11 subfields of physics - All agents exhibit a zero end-to-end callback success rate

Introducing Base44 Superagents. AI agents built with managed infrastructure, secured by default, one-click integrations, and 24/7 execution from the start. Everything is taken care of so you can focus on what your agent does, not how to get it running. That means no API keys to juggle, no config files, no security setup, and no maintenance. We handle all of it. Your Superagent connects to all the tools you already use in one click, runs on schedules and triggers, remembers context across sessions, acts proactively on your behalf, and keeps working around the clock. All from wherever you already are, WhatsApp, Telegram, Slack, or your browser. The AI agent everyone's been waiting for, with everything you need already built in. We're excited to get this into your hands, so we're giving free credits to everyone who comments and reposts in the next 24 hours.

5.4 sooner than you Think.

Neat idea to run a competition!

Kimi K2.5 tech report just dropped! Quick hits: - Joint text–vision training: pretrained with 15T vision-text tokens, zero-vision SFT (text-only) to activate visual reasoning - Agent Swarm + PARL: dynamically orchestrated parallel sub-agents, up to 4.5× lower latency, 78.4% on BrowseComp - MoonViT-3D: a unified image–video encoder with 4× temporal compression, enabling 4× longer videos in the same context - Toggle: token-efficient RL, 25–30% fewer tokens with no accuracy drop Here's our work toward scalable, real-world agentic intelligence. More details in the report 👉github.com/MoonshotAI/Kimi-K…

sf escape room called The Permanent Underclass and it’s just a room with a laptop and claude code installed

Grokipedia is promising but needs work. Lack of images, hyperlinks to its articles, small layout bugs etc. It's like a subpar Wikipedia (many articles are literally copy pasted with minor edits). Long term though it's critical it succeeds. Truth matters and its mission is noble.

LIVE5TREAM THURSDAY 10AM PT

Introducing Continuous Thought Machines New Blog: sakana.ai/ctm/ Modern AI is powerful, but it’s still distinct from human-like flexible intelligence. We believe neural timing is key. Our Continuous Thought Machine is built from the ground up to use neural dynamics as a powerful representation for intelligence. Thought takes time, and reasoning is a process. Biological brains inspire us with their complex neural activity, where neural timing is critical to intelligence. We’re exploring how to bring that power to AI. The Continuous Thought Machine (CTM) incorporates neuron-level temporal processing and neural synchronization, moving beyond current AI limitations. Our approach has two core innovations: (1) neuron-level temporal processing, where each neuron uses unique parameters to process a history of incoming signals for fine-grained temporal dynamics, and (2) neural synchronization, used as a direct latent representation to modulate data and produce outputs, encoding information directly in the timing of neural activity. Learn more about our approach: Interactive Report: pub.sakana.ai/ctm/ Full Paper: arxiv.org/abs/2505.05522 GitHub : github.com/SakanaAI/continuo…

The math reasoning dataset which has been used to train our winning model for the AI math Olympiad 2025 is now publicly available on Hugging Face huggingface.co/datasets/nvid…

The word “science” has two meanings, which are often confused. Traditionally, it stands for the rational pursuit of knowledge, while the contemporary use refers to institutionalized consensus seeking via peer review. The switcheroo seems to have happened between 1960ies and 90ies