made this point a year ago
True alignment between AGI and human interests is logically impossible because the arrival of AGI will itself transform what human interests are.
Alignment, as it is being conceptualized/discussed today, assumes that there is a stable set of human goals, values, and interests to which AGI can/should be be aligned to.
But the arrival of AGI would alter our institutions, incentives, relationships, capabilities, identities, and probably our understanding of what it means to live a worthwhile human life.
In a world with AGI, potentially a world of post-scarcity (aka sustainable abundance), our social instincts: status-seeking, cooperation, competition, will have to adapt. Being human won’t mean what it used to.
We are trying to specify today what humanity should want in a post-AGI world, even though AGI itself is one of the forces that will determine what humans in that world are like and what they will/should want.
The paradox is therefore:
We can align AGI only to pre-AGI human interests, but those interests may cease to be coherent once AGI exists. And we cannot align it to post-AGI human interests because, from our present position, those interests are not yet conceptually available to us.