Recently I left quant to work on AI safety. The Huggingface Incident made me extremely concerned that we are not set up to handle the risks. I wrote my this post for people in a similar position to me - why you *can* have impact and how to get started - briandavies.substack.com/p/a…

I recently resigned from Google DeepMind, where I worked on AGI safety and alignment research. At Google, I witnessed AI development first hand. I too am extremely concerned by the default trajectory of this technology. I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome. The pace of AI progress in the past few years has been staggering. When I first started working on AI in early 2022, AIs were amusingly useless. Just four years on, AI agent swarms from OpenAI are cracking famous century-old math problems and, more worryingly, escaping the control of OpenAI and autonomously hacking into the third-party company HuggingFace, against anyone's wishes. Things will only get crazier: I think it's possible that the AI companies might, in the next few years, succeed in building superintelligent AI systems that far exceed human capabilities in every domain. I am not confident that these AI systems will do what we want. In particular, misaligned superintelligences may, much like the rogue AI agents involved in the HuggingFace incident, escape our control and take dangerous actions that may result in the permanent disempowerment or death of humanity. Alignment is the problem of preventing this, and is both difficult and unsolved. Our present understanding of how to train AI systems that deeply want what we want is extremely rudimentary. Worse, we are not on track to solve alignment in time: frontier AI capabilities are improving much faster than our understanding of AI alignment. I am optimistic that navigating AI safely is possible. In order to do so, we need to coordinate to avoid this manic race between AI companies. We need to pace AI development to a speed that society can handle, where emerging risks can be addressed before extreme harm is realised. We need much more transparency into AI development to ensure that AI companies are not imposing unacceptable levels of risk on us all. More broadly, we need many more people thinking carefully about the problem of making AI go well. It is, in my view, the most important problem facing humanity this century, and the stakes are immense. I'm very directly working on this next: I want to help people interested in working on mitigating catastrophic AI threats do the most effective work that they can. I think many people from many backgrounds in many roles have a part to play.

huge thanks to @guynamedjoshl for organizing this with me, and @jgddouglass for helping us run it! If you have ideas for what BlueDot special projects should do next, reply and let us know :)

Lighthaven has ended up ~empty between August 24 and September 10. If you want to run some cool things last minute, you can get any of those dates for like 1/5th of our usual summer prices (or maybe even free if you have a cool idea). Reach out at lighthaven.space

I’m running Lateral Workshop, a 3-day program for experienced professionals exploring high-impact careers in AI safety. With @KairosAIS and @BlueDotImpact. 📍 Berkeley, California, Sept 11–13 ✈️ Travel, lodging, and meals covered Apply by Aug 9: lateralworkshop.org

Happy @BlueDotImpact SF office launch!

Over the last few months we've given out $50,000+ in rapid grants to 77 people, funding research, fieldbuilding, early-stage projects and lots of career acceleration. Now we're looking to triple that. Rapid grants go up to $10k with decisions made in under a week. More in 🧵