The fear - a real one - is that by the time we realize that the risk is existential, it may be too late. It’s like waiting to quit smoking until you get cancer.
However, here’s what will really happen and why slowing down private sector frontier AI will not really mitigate the risk.
Say we ban companies from pursuing RSI. That will not stop top secret government labs in the US, China, Europe, and even elsewhere from carrying on clandestine work in it. All we will have achieved is to remove the disinfectant of sunlight and public early warning signals. If RSI is truly dangerous, the only thing worse than having greedy (but often competent) corporations build it is to have power-hungry (and often incompetent) governments doing it. It will only accelerate the risk.