Aella brainstorming AGI bootstrapping plans...
"So first I offer some key engineers reverse gang bangs..."
One reason this question is tough is because it's subtly pointing at wrong assumptions.
This is why *I* was unconvinced about AI risk for so long - I was hung up on the 'but what are the specifics? Every specific chain seems so implausible, like any link in that story needs to have a flaw for the world takeover thing to not work.'
The *ingredients* are there. AI will almost certainly get more efficient (human brains are already WAY more efficient) and thus get better at running more cheaply and undetectably. You can already CRISPR print DNA. Nanomachines that convert air to matter already exist and are theoretically possible. We know human brains aren't even close to the theoretical limit of intelligence - neurons are super slow, for instance. We know humans are just happily and increasingly handing over control of their systems to AI. We know it's now easy to generate passes-as-real content, voices, impersonate people digitally.
But you try to string something together, and it's easy to imagine something going wrong. If *you* tried to like, manufacture a bioweapon, how do you know the CRISPR printer wouldn't flag and shut down? If YOU tried to make a bunch of fake videos to generate civil unrest, would it even work?
I became x-risk pilled when I stopped thinking in these terms, and started realizing that the specifics *do not matter*. Every time I try to think of a scenario, it is constrained by the fact I am thinking on slow human meat brain and I am an idiot. Sure, *I* can't figure out how to make the CIRSPR printer not shut down. *I* don't understand human psychology to know what kind of stuff would be effective. *I* cannot predict the extremely complex human systems well enough to feel confident that any plan theoretically conceivable by me could get past it.
But you need to understand that we are not even playing in the same ballpark when it comes to superintelligence. It thinks a thousand times faster than us, a thousand times better, and it has access to nearly all written language and video and code we've managed to shove into it. When we ask the question 'but give me a plausible scenario' it's like a child asking another child 'give me a plausible scenario of how this legendary 'adult' creature might somehow manage to kill our entire tribe of toddlers even though we are all united and armed with sticks.
Like. Okay. Sure. We can try to use our stupid meat brains to come up with stuff. We can point to the dangerous ingredients. But if you're still trying to piece together the exact storyline beore you find this convincing, you are deeply missing the point of the danger. You're thinking on the wrong plane entirely. You are not fully comprehending exactly the gravity of what superintelligence means.