AI can increasingly make progress on open problems like Navier-Stokes. Yet defining a new problem still relies on human research taste. Introducing ScholarCatalyst, a benchmark built from AI researchers’ firsthand accounts of what inspired their work.

Machine Studying on the Weaviate Podcast 🎙️ thanks @CShorten30! (yall can tell I had a great time there from the thumbnail!)

As I scale RL experiments using platforms like @tinkerapi (@thinkymachines) and @PrimeIntellect, one thing is becoming increasingly clear: purely on-policy RL is incredibly sample-inefficient, making it extremely expensive and unlikely to be the way forward for long-horizon agentic scenarios. We need to fundamentally rethink how agents learn from experience, beyond simply scaling on-policy rollouts. Excited to share one promising direction we've been working on soon!

it’s my nature to keep posts rather impersonal, so i have to work extra hard to tweet this sort of personal testimony below from time to time as i think the reminder in it around AI and the “value of human intelligence” is worth it as of the end of Oct 2026, i have not (yet - you may insist) had the *slightest* moment of questioning the impact of AI on the meaning of any of my work/skills - or even feeling any unusual shift beyond having an amazing (and rather often infuriating) tool believe it or not, but my ability to produce a completed artifact* (paper, model, open source software, lecture, …) and the effort, expertise, or time one takes to produce has actually barely reduced - despite comprehensive productivity benefits from AI the reason is that i essentially NEVER choose to work for a pre-determined outcome or some field-wide well-specified problem; i only work on artifacts i aim to perfect according to *my* judgment and where my personalized ambition obviously has no set limit and isn’t somehow pre-loaded onto AI better tools simply change the costs of my choices - i quite literally work on moving goal-posts if you will! when the fixed constraint are only time and energy, AI simply allows me to re-explore scope and make fewer (or different!!) tradeoffs or compromises, which feels almost no different! i’m not saying every job or every endeavor - including very advanced and historically prized ones! - looks like this today, but i’m offering: (1) at least one voice in this moment (of so many annoying loud FUD voices!) that it’s totally possible and i suspect extremely scalable and i predict will remain so for the foreseeable future! (2) why “someone like me” is indeed choosing happily to be in academia and not elsewhere! (again lots of FUD out here!, but where else do you have the freedom to keep seeking to create value for deciding and solving ~arbitrary ambitious/useful problems, establish credit for it, and train superstars who do the same?) my fear and worry in this moment is a lot less about Al or "the value of human intelligence" and much more about how concentration of power has a very reliable history of being very dangerous to large numbers of affected people! a VERY different problem imo *i guess certain of those took way longer than i expected but do stay tuned for a half dozen things on the short horizon…

AI is now taking on some of the hardest open problems that scientists have posed. But research is broader than solving problems. Can AI also propose research questions that experts find genuinely interesting? We built After42 to find out. Experts judged 45 of its 135 final questions genuinely interesting, across mathematics, statistics, theoretical computer science, and physics, with wide variation across subfields. The key idea: develop questions by trying to solve them. We also asked whether AI can judge which questions are interesting. We find that model judges' top choices align with expert consensus but do not consistently capture individual experts' preferences. We release the expert-assessed questions as an open resource for researchers to discuss, refine, and pursue, as a step toward integrating AI into the broader research enterprise. after42.org

What should a robot remember? Can it learn to remember better than we tell it to? Excited to share MEMI: Reverse-Engineering What Robots Should Remember 📈 97.0% end-to-end, 99.4% w/ low-level controller on RoboMME, nearly solving it for the first time 🧵👇

In language models, turning a generative model into a critic is simple: pass in the output and prompt for a score. Robot policies can't do this - they output physical actions & there's no built-in vocabulary to predict scores. So when it comes to RL on robot policies, we typically train a separate critic from sparse rewards - blind to everything the policy learned from the actions in the pretraining data. We think this has it backwards - can a critic learn from those actions the way the policy does? (1/n)

Memory is becoming a core part of agent harnesses/products (e.g. Instinct, @Muse, @cognition Devin, @PrimeIntellect Prime Agent, @NousResearch Hermes, @openclaw, @LangChain Deep Agents, @Letta_AI) -- yet despite every agent claiming to "continually learn", there's no benchmark to actually measure that capability. A big problem is that memory benchmarks today only measure recall (e.g. simple Q/A about past events). They don't measure if/how agents actually learn from experience to get better at work. At the same time, benchmarks for coding/knowledge work consist of many isolated, unrelated tasks - so they also don't allow for measuring cross-task learning. The context needed for a task is fully specified up-front (very unrealistic). In practice, both agents and humans work through sequences of many *related* tasks. Experience from past work provides context and learnings that apply to future work. This is why if you're a manager, you have to provide a lot more context when assigning tasks to a brand new employee or external contractor. Meanwhile, a great senior employee might not need any instructions at all. With FrontierMemory, we adapt existing benchmark tasks for coding and knowledge work so that instead of all the necessary context being provided up-front and within the task, it has to be learned from past experience. This allows us to compare how well different models and harnesses learn over time.

Benchmark paper: arxiv.org/abs/2610.02202 Laude only allowed registering up to 5 members, but the team is much larger @ohmyksh @Benjamin_eecs @dayoon12161 @RulinShao @seungonekim @gneubig @PangWeiKoh @achowdhery @AkariAsai @lateinteraction @YejinChoinka @gunheekim @chelseabfinn x.com/yoonholeee/status/2106…

ScholarCatalyst is part of Slingshots Four! Deeply “reading” the literature, i.e. knowing which earlier idea a new problem needs, takes years to get even okay at. Even the best systems do poorly on ScholarCatalyst. With @LaudeInstitute's support, we’ll train directly for this.

ScholarCatalyst // @yoonholeee, @ohmyksh, @chelseabfinn (@Stanford) — A benchmark for finding prior work that could inspire scientific research, and a retrieval system trained for this task.

Introducing Slingshots // FOUR: 14 projects across eight top institutions. Self-improving agent systems and harnesses, maximally-open models and training data, applications for research and decision-making – plus kernels and more. Say hello to the batch 🧵

Slingshots // Four is live! Another batch, another fourteen projects shipping open source artifacts that I’m excited about. See below for the full list. A few themes from the batch: 1. Harness time: This quarter harnesses shot up in visibility, with increased appreciation for the parts of an agent that aren’t the model and what they can give you. This batch includes a harness supporting persistent agency (Headlong); a harness supporting self-improvement via context management (ACE); a harness supporting self-improvement of the agentic harness itself (Exo); and a harness supporting multi-agent setups (dotagents). 2. Up and down the stack: There’s innovation to be had at every layer of the stack. We saw projects worth backing at the level of kernels (Thunderkittens); inference (M*); training (Marin); data (Atlantic-CC); evals (Decision Assistants); optimization (SkyDiscover); applications (Wagglehaus), etc. 3. Talk of the town: Slingshots continue to tackle timely problems at the frontier. Concurrent with Jev taking over X, an even faster type 1 decision model (CLM) came out of academia. With RSI on the brain of every big lab researcher, we now have a public literature retrieval benchmark grounded in authors’ firsthand accounts of their research process (ScholarCatalyst). And a new RL training method (Pedagogical RL) addresses head-on the question vexing RL practitioners of how privileged information can be used to discover generalizable behaviors, not just verify. Congrats to all the researchers in this batch and the previous three who continue to ship their research out into the world with open artifacts that can be adopted and built upon. Open science leads to better outcomes for everyone. And the good times will continue to roll – batch five is looking just as good. If you’re a researcher with a project that’s gaining momentum and ready for an adrenaline shot, shoot me a DM or apply at laude.org/slingshots. 👏 @a1zhang, @AlexKrentsel, @ankrgyl, @Azaliamirh, @bariskasikci, @bfspector, @chelseabfinn, @chipro, @Diyi_Yang, @dlwh, @drmapavone, @HazyResearch, @jackyk02, @jonasgeiping, @JonSaadFalcon, @jure, @kothasuhas, @lateinteraction, @martin_casado, @mertcemri, @michaelryan207, @MikeCafarella, @NickAJalbert, @NoahZiems, @nsagan271, @ohmyksh, @percyliang, @qizhengz_alex, @RussoMatthew, @samrmadden, @shagarw21, @ShashwatGoel7, @shulynnliu, @simran_s_arora, @souradipchakr18, @stuart_sul, @tianyu_li_, @tim_kraska, @WilliamBarrHeld, @yoonholeee, @YejinChoinka, @Stanford, @MIT, @LaudeInstitute, @CarnegieMellon, @UW, @UCBerkeley, @MPI_IS, @UWMadison

Question to post-training folks: Do we really have to do all that post-training on every base model just to find which ckpt would be the best coding agent? 🤔 What if we can spot potential BEFORE post-training? 🔮 Our method predicts future SWE-bench rankings with ρ=0.964 🔥 ⬇️

LLMs deployed in specialized domains, from enterprise codebases to law firms, dig through a mountain of unstructured context! Our new paper, Librarian Agents, asks: can an agent adapt by curating its own long-context workspace, anticipating queries before they arrive? 🧵⬇️

🧪Can AI agents take a scientific paper and write a meaningful follow-up to it? We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs: @LosAlamosNatLab @SandiaLabs @BerkeleyLab @AmesNatLab @PPPLab @Livermore_Lab

Applying to Stanford's CS PhD program this year? We are organizing the 2026 Student-Applicant Support Program (SASP) and would love to help with your application materials. SASP is a volunteer initiative run by Stanford CS graduate students. We provide feedback on statements of purpose and resumes for applicants from diverse backgrounds, broadly defined, before they submit their applications. We know how much work goes into PhD applications, and we hope an extra set of eyes on your materials can make the process a little easier. Please submit your materials by October 25 (Pacific Time). We plan to return feedback by November 20. More details and how to participate: cs.stanford.edu/sites/g/file… Please share this with anyone who might find it helpful! Questions are welcome at cs-phd-sasp@lists.stanford.edu. -- Bailey Trang (@baileytrangn), Bo Ai (@BoAi0110), Hermann Kumbong (@KumbongHermann), Kwon Ko (@hyungkwonko), Michael Paper, Swapnil Gandhi (@sw2pnil), and Yu He (@yuhe441)

The Pi API is officially in beta! 🤖 Many people have already been using it to fine-tune π models for their robots -- below are a few We’re hoping to expand testing significantly. Let us know if you are interested! Blog: pi.website/blog/api-beta Signup: pi.website/waitlist

I've been thinking a lot about making robot learning scalable. To me, that means learning on the job, and using the human help still needed efficiently. In Mulligan ⛳️, we focus each data-collection round on the initial states where the robot fails. This happens in deployment, with the robot doing the task and a person stepping in when needed. On 3 real tasks and 2,550 blind evals, it improves on uniform collection at the same budget.

Excited to see Meta-Harness featured in the 2026 @stateofai report! This was a great read; a lot of work points to text and code as places where genuine learning can happen. Full report: stateof.ai/