This is worth watching. It was an ill-posed question, but they could have answered much better.
What we should be doing is separate three classes of risk: Risks of AI as virus; 2: Risks of slowing AI; 3: Risks of users of AI.
1. The risks of AI being like a humanity-ending virus (I'd put this at 0, just like the real virus).
2. Many of the risks are in the risks of trying to slow AI: If global censorship/eye of Sauron is required to prevent weight sharing or god forbid limit compute, or even the insane e/a suggestions to bomb power centers, those are high risks of catastrophe. Like the risk of appeasement before WWII, by far the greatest risks we face IMO are mis-steps justified as precaution.
3: The risks of people *using* AI to execute a humanity-ending event. Like all risks that involve our free will and reflexivity across the wills of many actors, individual to state that can take any value we choose.
Like asking "What was the risk of WWI?".
It's didn't exist as some pre-destined Platonic number and could have been shifted to zero rather than lifted to as it was to one.
How to shift it is always the hard question.
Importantly, many attempts to set risks of continental war to zero turned out to be triggers that saw its actual probability set to 100%.
That's the real risk here too: "well meaning" (mostly e/a control freaks) people making a disaster where we could have paradise.
Granting one can't scientifically estimate this, is there an upper bound they'd defend?
I'll start: I'm confident the expected number of innocents my stats lectures will kill in the next decade is well below 100.
A\'s alignment research head estimated 800 million. Too high?