4/ How might AI kill all of us? Why can't we shut it down?
TL;DR: This is a difficult question to answer, partly because it involves predicting the future, and partly because the systems we’re worried about don’t exist yet. Think of playing chess against a grandmaster: you probably can’t predict the sequence of moves that will make you lose, but you can be confident that you are going to lose.
We’re in a similar situation with superintelligence. It could work at machine speed, absorb enormous amounts of information, and run around the clock without needing sleep. Millions of copies could work in parallel to create a perfect hivemind, making it feasible to give problems millions of hours of intelligent effort in days.
This problem gets even worse when we consider the power we will give AI. It will not stay digital: we’re already connecting AIs to labs, robots, militaries, and critical systems. AIs will also be superhuman at cybersecurity, be deeply integrated into every military, and be exceptionally good at persuasion. These capabilities would easily allow a misaligned superintelligence to manipulate people, seize control of critical systems, and cause catastrophic damage before we realize what is happening.
Even if this were true, why would we not be able to shut it down? We might not even realize anything is wrong! A superintelligence would likely be deceptive, and it would have every reason to appear useful while hiding behavior that would get it shut down. It would also be superhuman at cybersecurity, able to steal credentials, disable monitors, exploit vulnerabilities, or copy itself elsewhere while we tried to stop it.
To be clear, it is difficult to know whether catastrophe would occur through cyberattacks, biological equipment, military systems, or something nobody has imagined. But I encourage you to contemplate the ways that humans have dominated species of inferior intellect, and ask yourself this: if we build systems smarter than us, give them incredible power, and can’t control their goals, why would humans stay in control?