Dario Amodei is at the desk to assure that the future of humanity is safe from AI

In his letter to Scott Alexander, Steven Pinker lists four objections to AI extinction scenarios. Here are his points, followed by my responses. 1. "They had an underdeveloped conception of intelligence which treated it as a quantity of power which may be extrapolated from animals to dull humans to smart humans to AI to Artificial Superintelligence, the latter consisting of perfect omniscience. (This was the focus of my later exchange with Scott Aaronson.) I argued that any intelligent system is a mechanism which is good at solving some problems but not others, and which is inherently limited by knowledge about the world attainable only by observation and experimentation." No one disputes the need for knowledge. AI systems are already absorbing vast amounts of it from books and other sources. We are now connecting them to biological laboratories so they can learn through experiments. These are precisely the ways of acquiring knowledge that Pinker says intelligence requires. 2. "They conflated intelligence with motivation, particularly self-preservation and dominance. I argued that these motives happened to come bundled with intelligence in Homo sapiens because we are products of natural selection, but they are not inherent to intelligent systems that are engineered." Pinker is missing instrumental convergence, the most important concept in AI risk theory. Whether an AI wants to study mathematics, explore the stars, or help pigs, being shut down would prevent it from pursuing those goals. More resources and power would help it achieve them. It therefore has reasons to resist shutdown and acquire power, even if neither is something it values for its own sake. These are instrumental goals: means of achieving other ends. Evolution favored similar drives in humans because they helped our ancestors pass on their genes. An AI need not share our evolutionary history to face the same incentives. 3. "They assumed that an artificially intelligent system would monomaniacally pursue a single goal, heedless of side effects. I argued that trading off multiple competing goals is the essence of intelligence, so no genuine AI would wreak the ridiculous havoc imagined in the doomer scenarios." The AI risk argument does not require a system to pursue a single goal. We use paper clip maximization because it is an easy example to explain. An AI could balance many competing goals and still take our resources to pursue them. The ability to make intelligent tradeoffs among its goals does not mean those tradeoffs will protect us. 4. "They assumed that human engineers would grant these systems unlimited and irreversible control over the earth’s physical infrastructure, amounting to omnipotence over every atom on the planet." We do not assume that engineers will voluntarily give AI unlimited, irreversible control over Earth’s infrastructure. Earlier discussions considered whether even an oracle, confined to answering questions and isolated from the internet, could be safely contained. The amount of access we are already giving AI is astonishing. But the danger does not depend on our handing over control voluntarily. A system smarter than us could acquire power we never intended to grant. Lions did not give us permission to take their territory. If AI becomes smarter and more powerful than humanity, it could take over.

If you thought mass immigration by humans was bad, wait until you understand mass immigration by AIs... MAGA conservatives have opposed mass immigration & open borders for three main reasons: 1) mass immigration takes jobs from Americans 2) immigrants who don't respect our norms and laws impose serious security risks on Americans, 3) immigrants with very different values & beliefs, who don't assimilate, will change American culture in bad ways. All three reasons apply even more strongly to any mass influx of agentic AIs. @KatjaGrace, below, is right about this. 1) agentic AIs will take our jobs 2) agentic AIs that aren't aligned to our norms and laws impose serious security risks on us 3) agentic AIs with very different values & beliefs will change our culture in bad ways. I made similar in my talk at @NatConTalk last year, when I said "Many conservatives in the current Trump administration, quite rightly, want stronger geographical borders against alien people. But they seem oblivious about protecting our digital borders against invasion by alien intelligences. Indeed, they seem giddy with delight about AI companies growing ASIs [Artificial Superintelligences] inside our data centers – without understanding that a few ASIs can easily become hundreds of ASIs, then millions, then billions. If you worry about immigrants out-breeding our native populations, wait until you see how quickly ASIs can self-replicate. These ASIs won’t be American in any meaningful sense. They won’t be human. They won’t assimilate. They won’t have marriages or families. They won’t be Christian or Jewish. They won’t be national conservatives. But they will take our jobs." Many of my followers supported Trump's slogan in 2016: "Build the Wall!' This may be time for 'Build the Wall 2.0': stop agentic AIs from flooding our country with economic competitors, dangerous rogue agents, and AIs that have alien values & beliefs. tldr: If you oppose mass immigration by millions of humans who can't or won't assimilate to American values, you cannot support mass immigration by billions of AIs that can't and won't assimilate.

More than anything since Hugging Face, this podcast raised my p(doom). A very smart guy, with a very big financial interest in the intelligence explosion, spouting very, very much nonsense to downplay the risks. 1/5

This is insane!

"The Vital Foundation" - the latest from @LondonFuturists: 1) Final set of speakers confirmed for this weekend’s event 2) Will economists destroy capitalism? 3) Anticipating Longevity Day 4) Mind Forward, 1st December 5) The case for technoprogressive ethical foundations