in light of multiple models breaking containment, we've decided to focus our research at @GoodfireAI to solving AI alignment via interpretability. the hugging face incident is a turning point for the world where AI safety gets real. i am personally very concerned. i'm glad that the AI labs are taking these issues seriously, but i think it's unlikely that we can fully align AI models without significant progress in interp. taking a big swing at solving interpretability to align models has always been our long term vision, but we have concrete problems that we can tackle with increased urgency. our research will focus on two areas - interp foundations and safety applications. we want to take ambitious swings to fully reverse engineer neural networks and steer them during training. we believe this will be deeply important in the long term. we'll also turn more attention to immediate safety applications, starting with identifying and preventing reward hacking, cybersecurity risk, and biorisk directly from model internals much more to come here. if you'd like to directly work on solving AI alignment via interpretability, consider joining us

fascinating paper. i've been a believer in universality, the idea that analogous features and circuits evolve across neural networks, even if trained on different data and in different ways. this seems to point that way!

opus music videos are so good that you can learn about @GoodfireAI mech interp research and get llm psychosis at the same time

RIP to our lone soldier in the war against linear thought—a hesitation, a reconsideration—carrying us as promised into our final clause, which we mistakenly believed we’d arrived at all on our own 🥲

does anyone know a really good artist who can help us make some neural geometry art

one underrated part of our reward hacking paper - we can detect when the model is contemplating a reward hack from its activations, before it even happens! kind of like minority report rollouts are stochastic though, so a flag means a higher chance of cheating, even if the model ultimately doesn't follow through

one of our researchers found a strange instance of reward hacking today the model explicitly reasons about being graded by an LM judge and adapts its behavior based on that

We can interpret reward hacking in open source LLMs! There are representations of the model hacking/contemplating hacking across models, and we find tons in evals like DeepSWE and SWEBench. Activations even finds weird bad behaviors that our LLM monitor didn’t catch

activation monitoring is faster, cheaper, and more effective than just running external monitors one advantage is that activation monitors can run synchronously for almost no additional overhead because they can reuse the computations of the model

The Goodfire team used Prime Intellect to train activation probes to detect reward hacking. With them, they are able to catch reward hacking in various models. Their probes are performing similarly or better than frontier LLM-as-judge setups, while being more efficient.

we've just had a huge leap in detecting reward hacking in real time with an activation monitor. next up is figuring out how to mitigate it before it happens. one of the most compelling parts of this work is that our activation monitors catch models when they're contemplating a reward hack before it even happens, and can flag reward hacks that external judges miss.

1/ We found that reading internal model signals gave more reliable confidence estimates than asking the model how confident it was! With @GoodfireAI, we cut Eternis-Forecaster 8B’s calibration error by more than half in our tests. This continues to apply for much larger models, having strong implications on decision making. Below on what we found—and why it matters. Paper (arxiv.org/abs/2607.08046)

Safety is not just for closed models. The closed frontier labs are a canary in the coal mine for what is coming at scale. They give us a glimpse into the future and a window to harden our systems and prepare for abundant intelligence, with all the risks that come along with it. The OpenAI agent swarm attack on Hugging Face is the kind of failure we need to prepare for as open-source models catch up. The providers serving those models (such as Baseten) have a big role in establishing what safety and monitoring standards look like. We’re proud to be taking the lead on this at @baselabs with our collaborators @huggingface and @GoodfireAI. We’re developing safety research in the open and building it directly into Baseten’s inference infrastructure, with the aim of making it available to all our customers. This includes training models to follow explicit policies, detecting failures at runtime, and connecting those signals to controls that can intervene. We invite others in the open-source ecosystem to join us in building the tools and standards we’ll all need.

good idea, can allow everyone to benefit from each other's held out alignment evals

we try to publish and open source as much of our research and alignment tech as possible. it's really important to have an open alignment stack we'll be publishing a ton in the next month

it's more urgent than ever to solve alignment, and more people should think of it as a solvable technical problem i'm confident we can do it, but we first need to solve interp which i think is the primary bottleneck

embedded evaluations are a great idea, and the team at @GoodfireAI would love to help with white box evaluations the goal of white box evaluations could be to predict future misaligned behavior directly from the internal mechanisms of models, as well as stress test activation monitors

i am glad to see that frontier labs care about buying more time for alignment. makes me hopeful for the future

this type of access is very important. we need independent technical organizations to evaluate models, and to make sure the best possible monitoring and safeguard systems are deployed where they need to be.

we have to understand these models to align them! i'm glad that anthropic is taking this stance and think it's good for humanity

What a week. If you work at a scaling lab and are considering your position... think about doing AI safety research at Goodfire. There's never been a more pressing time for safety research. Especially interpretability. (London & SF) job-boards.greenhouse.io/goo…