Recently rewatched Oppenheimer, and there was a line that stuck with me:
Kitty Oppenheimer to JR Oppenheimer: “You don't get to commit sin, and then ask all of us to feel sorry for you when there are consequences.”
Different context, obviously, but sadly, the "I told you so" will never be enough
While I assume we will continue racing ahead on the AI buildout due to game-theory and political considerations that are completely out of the hand of lab researchers I am more sympathetic to the safety concerns than most and actually don’t think regulatory capture for financial reasons is the primary motive.
That said, I also think the belief that “if we get there first we can safely birth it” at Ant was poorly thought out as it clearly didn’t properly contemplate the societal, geopolitical, supply-chain or general economic dynamics that would surround arriving at near ASI in a race environment.
I also think that while most researchers *believe* they are being max morally good to some degree there is a motivation for preemptive indemnification where they get to continue reaping the monetary, intellectual and status rewards of being at the epicenter of birthing the machine-god while also being able to insulate themselves from the inevitable repercussions or mass-casualty event by saying “hey we warned you and you forced us to keep going”.
I take a more sociopathic view on this last part.
If you’re not ok with taking the tail-risk that you’ll end up in jail because GPT8 killed a bunch of people…then quit.
And if you don’t quit then be cognizant that you are signing up to go to war. And to the victor the spoils.
But if your general fucks up you will get slaughtered by the masses as they look for a throat to choke.
What you can’t do is offload the tail risk to society broadly while monetizing the right tail.
Sam thought he could do exactly this by not taking equity and attempting structures like the public benefit corp.
But that is insufficient because he is still reaping the intellectual and status rewards previously mentioned.
I am not saying this is fair. I am saying this is by far the most likely construct within which you’ll operate.
Luckily for you the IPO is a great opportunity to monetize your efforts so far and then sail off into the sunset.
But know if you stay at the labs you are choosing to potentially sacrifice yourself in a moonshot towards greatness for humanity.
And there is no life raft.