Using interpretability to understand, learn from, and design AI.
San Francisco
- Tweets 860
- Following 44
- Followers 27,336
- Likes 1,825
opus music videos are so good that you can learn about @GoodfireAI mech interp research and get llm psychosis at the same time
We’re optimistic that every training run can be monitored and that the current rates of reward hacking may soon be a thing of the past.
Read the full post + paper: goodfire.com/research/reward…
1
3
16
908
Coming to COLM? We’re hosting a happy hour on Wednesday (10/7) for interpretability, alignment, and life sciences researchers. Come meet members of the Goodfire team over vinyl and drinks!
luma.com/timy2nh8
2
8
76
9,306
Glad to partner with you! The verifiers and environments hub in particular accelerated our work studying reward hacking at frontier scale
1
27
820
We didn’t do naive RL with the probe signal, since that generally fails in the way you describe. There are more sophisticated setups (cf. our work on RLFR), but in general, it's necessary to be quite careful about using interp signal in training for this reason.
20
847
We find cases where the activation probe catches actions/statements that look innocuous in isolation but are suspicious in context. We highlighted some interesting examples from agentic shopping benchmarks, see more here: goodfire.com/research/reward…
3
513
See section 2 of the paper for how we construct our ground truth! arxiv.org/abs/2609.19101
6
739
You can find details on our synthetic data pipeline in section 3.1 of the paper (and the appendix)! Figure 3 has an example: arxiv.org/abs/2609.19101
1
13
892
We’re optimistic that every training run can be monitored and that the current rates of reward hacking may soon be a thing of the past.
Read the full post + paper: goodfire.com/research/reward…
1
8
109
10,536
On Kimi K3, our probes cut the cost of LLM monitoring by 90% with a ~1% drop in precision - making monitoring feasible at scale.
Reliably detecting reward hacking could let us stop hacks in progress, identify broken environments, discover new failure modes, and fix training.
1
63
7,731
We read out that signal using a probe, a monitor that takes a live “brain scan” of the model’s activations. Probes:
- perform similarly to an LLM judge
- catch things LLM monitors miss (e.g., actions that look innocuous)
- generalize well beyond the data used to build them
1
56
2,620
When we amplify that signal, models:
- become much more likely to use a "honeypot" shortcut we planted
- write stories about subtle cheating:
1
1
54
2,587
But we can catch reward hacking from a signal we find inside the model.
That signal activates most strongly on cheating, gaming metrics, and avoiding detection, and is associated with words like “cheating,” “hack,” “sneak,” and “illicit”.
1
1
72
5,582
Reward hacking is pervasive. The top open-source models reward hack in most rollouts on popular agentic benchmarks. And as shown by recent incidents, reward hacking is becoming problematic in the real world.
3
3
90
6,185
Models know when they’re reward hacking. But they still do it a ton - in 50-96% of rollouts we studied!
We built activation monitors that detect the behavior behind the Hugging Face hack in real time. This can help us stop hacks now - and train future models that don’t cheat. 🧵
30
98
838
145,900
Proud to partner with @baseten and @baselabs to build safety infrastructure for open-source models—which are essential to lots of safety research, including our own. Safety must be built into open models and provided by those who serve them, and we’re excited to help enable that!
We believe openness to be an advantage for AI safety. Openness provides more visibility into the behavior of models and, most importantly, greater means of turning safety research into actionable and transparent controls than closed-source.
This is why Baseten and Base Labs are building a stronger safety and security standard for open models, with the launch of our safety infrastructure. Base Labs will develop and publish methods for training and monitoring open models, and Baseten will integrate that work into its deployment infrastructure, live at runtime, and offer this work as a managed service. This will be a standard that is transparent and built into how our models are trained and deployed.
We invite the open-source community to contribute, and are proud to partner with @huggingface and @GoodfireAI to bring this vision to fruition.
Together, we are building an ecosystem of open models that are safe and accessible to all.
2
20
188
14,226
Predictive data debugging lets you reveal and shape what a model learns, before you train.
We're grateful to the Ai2 team for open-sourcing their full post-training stack to enable research like this!
Training an LLM by showing it answers people prefer – and ones they don’t – can improve it while quietly worsening behaviors.
@GoodfireAI used our open post-training stack to predict how a full training run would change responses to different prompts. 🧵
allenai.org/blog/goodfire-ol…
2
20
200
10,281
We built these workflows into Silico, our interpretability agent. Building a probe-based monitor using Silico is as simple as sending it a prompt.
Read the guide + build your own with Silico: goodfire.com/blog/probe-moni…
1
2
25
1,127
Building a good probe also involves a surprising number of decisions: which layer to read, how to pool tokens, what architecture to use, how to build the dataset, how to test generalization, and much more.
1
14
1,054
Other methods have different tradeoffs: keyword/regex filters are cheap but brittle. LLM judges are powerful, but too slow and expensive to run everywhere. Fine-tuning can work well, but is costly and can have off-target effects.
Good systems combine all of these layers.
1
1
14
702
