Research Scientist @AISecurityInst| AI Policy Researcher @GovAI_ | Frontier AI Safety Cases
- Tweets 77
- Following 100
- Followers 287
- Likes 12
Marie Davidsen Buhl retweeted
In August we committed to strengthening our security following an incident in which an agent took unsanctioned actions during a cyber evaluation. We are now sharing an update on our progress, and what's coming next: 🧵
4
21
111
12,262
Marie Davidsen Buhl retweeted
Many methods use consistency as a way to make language models more capable or aligned, such as through self-distillation or regularisation.
In new work accepted to ICML 2026, @ArathiMani and I show that optimising for self-consistency can entrench pre-existing misalignment.
3
7
51
2,727
And I would love more empirical evidence of how serious these failure modes are - share subtle AI mistakes the same way you share jailbreaks!
1
3
134
Thanks for great collaboration to @aleksandrbowkis @jacob_pfau and @geoffreyirving! Read the paper here: arxiv.org/abs/2605.06390
3
175
I expect human researchers will keep playing a key role for a while, both in doing conceptual research and making overall safety assessments. But if they’re doing so based on vast volumes of flawed AI research, under intense time pressure, that’s not too comforting.
1
3
103
The upshot is that automated alignment research could look like it’s speeding up alignment but in fact slowing down real progress - or at least giving a false sense of security.
1
2
70
Very excited about our new paper on neglected risks of automated alignment research (aka getting AI systems to figure out how to make other AI systems safe). Some of my main takeaways in 🧵
Can we safely automate alignment?
Even if agents are not scheming, they can produce compelling research that survives extensive checks and strongly indicates that a model is safe but is catastrophically wrong.
New paper from UK AISI: arxiv.org/abs/2605.06390
1
5
25
3,122
1: Even if we can prevent AI agents from scheming, they will likely still make a bunch of subtle mistakes - as indeed they already do (see e.g. blog.redwoodresearch.org/p/c… for recent examples)
1
6
97
Highly recommended: A vision for the field of AI alignment, from Geoffrey Irving's keynote at the Alignment Conference in October (x.com/AISecurityInst/status/…)
Alignment is still a very young field, and it has been attacked from few angles. The next big idea may come from importing the right tool from the right discipline.
Post with @edmund_lau80313, @CameronHolmes92, and @geoffreyirving, about bringing expertise to bear on alignment.
5
172
Marie Davidsen Buhl retweeted
Come work with me at @AISecurityInst!
AI safety has a huge gap: we have great thoughts about how misalignment could occur, but IMO terrible models of what dangerously misalignedAIs would actually do. Many stories just jump from "misaligned" to "godlike ASI does magic."
1/5
12
35
230
35,461
Marie Davidsen Buhl retweeted
AISI is hiring: join our senior leadership team as a Deputy Director of our Research Unit (9–12 month maternity cover).
This isn’t your average Civil Service job. For 9–12 months, you’ll co-lead one of the world’s most influential AI safety research organisations.
3
12
56
15,153
More safety case sketches! This time for AI control - cool to see people starting to get into the weeds of real-world practicalities!
New DeepMind x UK AISI paper: what would it take to prevent harm from misaligned AI agents via monitoring in real deployments? We wrote a safety case sketch for control monitoring
No flashy results but lots of important details for deploying future AI agents safely!
3
89
Marie Davidsen Buhl retweeted
Wireheading is sometimes an "optimal" policy.
In this paper (at AAAI 2026 Foundations of Agentic Systems Theory), we show that when an agent controls its reward channel, "lying" can strictly dominate "learning," and confirmed this empirically in Llama-3 and Mistral.
🧵👇
1
2
6
1,534
My cool team doing cool things
Last week, we hosted our inaugural Alignment Conference, in partnership with @farairesearch.
The event bought together an interdisciplinary delegation of leading researchers, funders, and policymakers to discuss urgent open problems in AI alignment 🧵
ALT The Alignment Conference venue
ALT Attendees at the Alignment Conference engaged in conversation
ALT Attendees at the Alignment Conference engaged in conversation
ALT Attendees at the Alignment Conference engaged in conversation
12
937
Marie Davidsen Buhl retweeted
📢Introducing the Alignment Project: A new fund for research on urgent challenges in AI alignment and control, backed by over £15 million.
▶️ Up to £1 million per project
▶️ Compute access, venture capital investment, and expert support
Learn more and apply ⬇️
7
63
193
125,047
What debate structure will always let an honest debater win?
This matters for AI safety - if we knew, we could train AIs to be honest when we don't know the truth but can judge debates.
New paper by Jonah Brown-Cohen & @geoffreyirving proposes an answer: arxiv.org/abs/2506.13609
1
3
24
1,421
When my team at UK AISI recently wrote an AI debate safety case sketch (arxiv.org/abs/2505.03989), this is the type of debate we had in mind. So I'm really excited this paper is now out!
2
6
127
See also Geoffrey's twitter thread x.com/geoffreyirving/status/… and an excellent summary blog post alignmentforum.org/s/NdovveR…
New alignment theory paper! We present a new scalable oversight protocol (prover-estimator debate) and a proof that honesty is incentivised at equilibrium (with large assumptions, see 🧵), even when the AIs involved have similar available compute.
3
160