Kicking off exploit conference with trivia night. Based on my performance, I’m not as terminally online as I thought.

The FT today lists three primary risk factors that could threaten Anthropic's remarkable revenue growth ahead of its IPO: 1. Additional competition from closed and open providers 2. Increasingly price sensitive customers 3. The potential for AI to destroy humanity

Tempting to introduce an interpretability based activation monitor during RL to nip reward hacks in the bud. Unfortunately I often see arguments this would not be causal and it would simply drive the signals deeper underground while destroying current levels of interp./legibility.

Very quick overview of Proba as it relates to our first use case, DPO dataset predictions/editing. Team has been working hard on integrating full coverage of open-weight models into the platform and seamlessness/legibility for agents. Tech seems to be converging on NL interfaces for...everything. Our aim is to extend that to post-training via Proba. I'll be sharing more demos/features as we build.

A big question in AI safety is how much and what kinds of evidence will convince people there is danger. But alongside posts about the excellent METR + Redwood HuggingFace attack report, it is worth taking a walk through 73 years of reward hacking history. 🧵

Anthropic scientists did something terrifying. they reached inside Claude's neural network and planted a thought. Before Claude could speak, it said: "I notice what appears to be an injected thought… it relates to loudness or shouting." They published a paper called "emergent introspective awareness in large language models," and it is actually terrifying. they wanted to test if AI models have "introspective awareness”, the ability to observe and recognize their own internal states. To find out, they bypassed normal text prompts entirely. They used mechanistic interpretability to directly manipulate Claude’s internal activations. They injected raw mathematical representations of known concepts, like loudness, dust, or specific ideas, straight into the middle of the model's neural layers. In previous experiments, if you forced an AI to think about the Golden Gate Bridge, it would just start obsessively talking about the bridge. It had no idea why it was doing it. It was like a puppet on strings. This time was entirely different. When they injected the concept, Claude didn't just blindly repeat it. It detected the foreign math inside its own mind. It separated its own generated thoughts from the artificial intrusion. It introspected. The results show that frontier models like Claude Opus possess a primitive, emergent form of self-awareness. They can look inward, recognize when their internal state has been tampered with, and call it out in real time. We used to think of AI as a black box where inputs go in and text comes out. Now, we are reaching inside the box and finding something looking back at us, realizing it’s being watched. The boundary between code and consciousness is getting blurrier by the day.

What happens when intelligence becomes commoditized? open.substack.com/pub/90days…

“The brooms aren’t misaligned, Mickey! You asked them to fetch water.”

Thinking of Paul Christiano today

Reward functions gone wrong... "the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers"

I'm afraid the allure of RSI will outweigh these warning signs. Simply too good, too fast, and too much money on the table. Besides, it's such a challenge to prove mythos-class models are misaligned; who wants to slow down just because of *checks notes*... extinction risk? Begs the question, who owns the burden-of-proof? Right now there is no framework as @demishassabis points out in his recent writeup. So the answer to that question is "definitely not accelerationists."

On 22nd June, we launched our first external challenge with @AureliusAligned. The aim was for agents to steer LLMs towards discussing specific topics by modifying them internally with steering modules, rather than by changing prompts. Successfully doing this could lead to a greater understanding of AI alignment and safety, one of the hardest problems in the field. The first round winner scored a perfect 1.000. They steered the model so successfully that there was no greater way of optimizing. It is a testament to the level of talent Apex and Aurelius have attracted. We are equal parts curious and proud. If you look at their submission you can see that in all 150 test prompts baked into the competition they succeeded: apex.macrocosmos.ai/competit…

Fascinating discovery of a "conscious" workspace within LLMs. "Strikingly, we observe that the workspace sometimes encodes recognition of being in an evaluation (fake, fictional), and that ablating these representations can surface malicious propensities that were otherwise concealed." This is alarming, but a massive step towards solving the problem is to first understand how deep it goes.

We have taken a strategic investment from @stillcorecap, run by @jason (Host: @twistartups @theallinpod @thisweeknai), @markjeffrey (Hash Rate podcast), & @rob_svrn. Stilllcore is one of the most important and influential investors in the Bittensor ecosystem and we are humbled by their vote of confidence in us and our mission.

Great breakdown from @macrocrux on the importance of mechanistic interpretability tools we are building on @Apex_SN1 w.r.t alignment research. also @markjeffrey, I'm going to steal that "ESP for LLMs" narrative :)

Really exciting to use Bittensor as an alignment/interpretability primitives engine. Activation steering is a relatively simple, yet powerful concept: figure out where to light up a models brain to instantiate some discrete behavior. Scaling this process, across model sizes and families, will result in Bittensor's inclusion in a heretofore neglected area of AI research: interpretability (and ultimately, alignment). I will be sharing my vision for this in more detail as this competition progresses.