Is opposing AI takeover speciesist? Which successors are good? What perspectives is the AI space missing that make it confused about these questions? New post out, that completes my series on alignment & succession. Thread -->

We’re releasing privacy-preserving, transparent and verifiable safeguards in Tinfoil Chat. The trending narrative says safety requires sacrificing individual privacy and freedom. We reject this and demonstrate that both can coexist. We also reject the absolutist argument that once any scanner exists in a privacy-preserving system, mass surveillance is the only outcome. We believe that preserving individual freedom and privacy long-term requires a responsible policy for responding to abusers who could ruin the promise for everyone.

There are two ways AI progress could go very badly and that we must avoid. First, we could lose control of the future to AI. This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people. To ensure that, we need ways to ensure that alignment and safety techniques stay ahead of progress in model capabilities. Second, we could end up in a world with too much concentration of power. If an extraordinarily powerful AI is used by one person or company to impress their worldview onto everyone else, the results could be extremely dystopian. Avoiding these two threats requires walking a narrow middle path; for example, one country could gain too much power. Another example is one lab ending up with too much power.

Since about 2023 my rough sense of outcome buckets for humanity by likelihood has been: 10% foom&doom: at some point AIs climb very aggressively in capabilities and we fully lose control over them due to their actions & planning 40% bad selection: selection processes driving bad or very suboptimal outcomes due to important good things (like free societies, or humans in positions of power or economic actors) being outcompeted 10% stagnation: late in this century, as result of war or civilizational stasis & decline in both West & China, tech doesn't advance much 25% weird: deeply weird / hard to judge / mixed, would take lots of time to form an opinion on 15% obviously good: flourishing future by our current lights These ratios have been pretty stable, and I’ve found it hard to get much evidence to greatly lower or reduce any bucket. Compared to many I have larger error bars and feel more averse to “worldview distribution collapse”. Evidence from the OpenAI HuggingFace hack and considerations of RSI mechanics have made me update. I expect to settle around 14% foom&doom, with most of the increased mass there coming from the more extreme side of the bad selection bucket. You can have gripes with assigning numbers to things but I would encourage people to make their views more comparable and actionable and quantified, to help create common knowledge as we head into fast-changing turbulent times

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.