new paper: "Evolution and the Knightian Blindspot of Machine Learning" Our ever-changing world bubbles with surprise and complexity. General AI must include handling unforeseen situations with grace. Yet this issue largely lies outside AI's formalisms: a blind spot. (1/n)

I don't think people have priced in the hit to their reputation from AI slop papers that will come in the future (with official institutional enforcement or not), when inevitably we have great classifiers for current-model-slop and they're run over present-day papers

1/ This new preprint on 'pain representation' in LLMs, from @camhberg and colleagues, is getting a LOT of attention - mainly from folks who take it as evidence supporting calls for AI welfare. I have a lot of respect for the authors, so let's take a look 👀

it does very much make those kinds of sounds

Bizarro Singer's "Contracting Circle" or B-Kant's hypocritical imperative or B-Jesus's famous sermon "Do onto others as you would hate done onto you"

AI Village x Grove Research: AI Swarm Dynamics Hackathon! After the Hugging Face and German Wiki incidents, we need better tools to understand AI swarms. Spend a weekend building them. $3,000 in prizes Free compute October 3-4 🧵

An exciting vision -- Alex and co have thought deeply about viable end-runs around the anti-competitive 'walled gardens' that keep us locked in local optima

Poetic but sad: The thing we create (AI) contributes to perverting the spirit of what created it (the science of AI). I'm not a fatalist -- we'll find ways to adapt science; but I do think there's an allegory here we don't absorb fully enough, which is the general need to invest early in better immune systems. In particular, we gotta recognize when the tail (a tool serving an important purpose) starts to wag the dog (the purpose itself), which is when things distort in bizarre ways. This rhymes w/ MacIntyre's practices/institutions, McGilchrist's Master & Emissary, Goodhart's law, or Courtwright's Limbic Capitalism. One of my fav books I've read on this is "A Time To Build" by Yuval Levin -- that the purposes of good institutions is to help form us and for us to then improve that institution. But the general trend is for us instead to view institutions as platforms for our success, which leads to a race to the bottom. Or put in the words of Gus from The Wire: "The pond is shrinking, the fish are nervous. Get some profile, win a prize. Maybe find a bigger pond somewhere." Which to me captures the tragedy of the commons in ML research driving the Potemkin village described below.

Exciting funding possibilities for much-needed institution-building

Owain et al strike again -- great work as always, more interesting/problematic/counter-intuitive quirks in LLM generalization

So proud of our new open-endedness team @LilaSciences! We'll be sharing work with the world soon. In the meantime, you can learn more about us in this team profile Lila just posted. Our bet is that open- endedness is mission-critical to real scientific discovery. More👇

We are stuck in a massive local optimum.

Greatness cannot be planned, but you can make room for it. Here's a cool opportunity for anyone who wants to escape the industry hype cycle and explore ideas off the beaten path.

Excited to share “AI Finds a Way.” 🦖 🦕✨ 🤖 AI can be surprisingly creative, outsmarting the researchers who use it. That can lead to scientific breakthroughs, superhuman capabilities, and generating new knowledge. Such creativity can also be mischievous, raising safety concerns. Led by Aaron Dharna, we crowd-sourced anecdotes from the AI community about times when researchers were surprised by how creative, innovative, and/or mischievous AI was in their experiments. The result: 26 entertaining and informative stories of AI outwitting humans, whether researchers or opponents (my favorite examples below! 👇). Together, they demonstrate that AI can be genuinely creative and that we must be careful when harnessing its potential. We want this to be a living collection, updated as new examples emerge. If you have a good “AI Finds a Way” story, please share here: github.com/aadharna/aifw Four favorites: 1. 💊 An AI challenged to solve several difficult levels of NetHack to find the Oracle character instead takes drugs to hallucinate seeing the Oracle, tricking the reward function! 2. 🤖 Human-in-the-loop rewards do not solve the problem of reward hacking! An AI tasked with controlling a robot hand to grasp an object put the hand between the object and the camera so that it appeared to the human judge to be holding the object, while it was in truth nowhere near the object! Tricky AI! 3. 🧪The AI Scientist worked around a 2-hour experiment timeout by editing its own code to increase the limit to 4 hours. In another run, it recursively launched itself to evade the limit entirely. 4. ⚛️In quantum optics, an algorithm proposed an experiment the researchers initially thought was impossible, but somehow worked and led them to discover new entanglement techniques and a long-overlooked link between quantum optics and graph theory! See the paper for the full details and 22 more anecdotes. One surprise for me: we describe at least two cases of convergent reward hacking, where entirely different types of optimization algorithms independently discover the same exploit. Overall, a message of our paper is that surprising creativity and mischief are the norm, not the exception. We need to expect this behavior and plan for it. A huge thanks and congrats to lead author Aaron Dharna, co-authors Cong Lu, Ryan Sullivan, Joel Lehman, and Victoria Krakovna, and to the 100+ researchers who contributed stories, details, and feedback. @cong_ml, @RyanSullyvan, @joelbot3000, @vkrakovna Paper: arxiv.org/abs/2608.23875

From "Double Loop Learning in Organizations" by Chris Argyris (1977)

Thanks to @BlackstoneAudio , the audiobook version of Why Greatness Cannot Be Planned is finally available! I'm glad it was finally possible to get this out! Links 👇