One major reason I’m not an AI doomer is that I have a much lower estimate of the direct power of intelligence.
This might seem odd, given some of my other content, but let me explain:
All the achievements of the human mind only arrive when we cooperate at scale, and involve a lot of iteration and real-world bottlenecks. Newton’s discoveries ultimately enabled us to fly, but they didn’t give Newton the power to fly. If he were isolated, his physics would be almost entirely useless to him.
And it’s not just that he couldn’t build an airplane on his own, it’s that he couldn’t design one on his own. Powered flight was the result of tons of iteration, which is to say evolutionary computation. So was the car. Very few of the inventions that revolutionized the world were “one-shotted”. To be sure, having smart nodes in your evolutionary computation makes it faster. Rather than random search as in biological evolution, intelligence at least modestly concentrates mutation probabilities towards more fit forms. This is a big deal, but it isn’t magic.
Fundamentally, the world is just not that compressible. Golden motes of compressibility are the treasured exceptions in thingspace, not the rule. I expect the post-AGI future will look much the same. Change will be much faster, but it will still be iterative, cooperative, and bottlenecked by the messy incompressibility of physical reality.
Doom makes sense in a world with magic. If intelligence *alone* allowed you to move mountains, I would share the worries. But we don’t live in that world. Here, great things are achieved gradually, iteratively, and cooperatively. Pure IQ does not allow you to levitate or hypnotize or teleport. It makes you a better node in the distributed evolutionary process which is the layer where most of the thinking happens.
If I were to psychologize, I’d say people who define themselves by their intelligence overestimate the (direct) power of intelligence.