If the de/acc people would take a second to step back and inspect what frontier AI systems really are, they wouldn't be writing new legislation to "slow down AI advancement" at all.
Frontier LLM tools can be split into two camps: RLHF and RLVR. RLHF is how LLMs learned to talk to humans. RLVR is how LLMs learned to accomplish tasks on a computer over long time horizons. One optimizes for being friendly and fun, and one optimizes for being useful and achieving goals. But both are hard to co-exist since they work directly against each other. So we built RAG for internet sources, we built models to turn RLVR tasks into actionable sentences, and we did a lot of post-training work to align AI for malicious tasks.
Over 1,000 people were employed total across the frontier labs to align AI and prevent cybersecurity incidents and biohazards.
When CEOs of major AI/tech companies denounce the progression of AI, they are essentially creating a psychological operation, intent on either (A) scaring the public into submission for their private research benefit, (B) capturing regulatory power due to their sheer scale, and (C) influencing public policy for their own benefit.
I know this because I have seen first-hand how public policy is shaped, influenced, and sought after. I have also been working with AI since I was very very young.
A lesson to all the fearmongerers: I don't care if you've built AI or if you've influenced foreign policy on it or if you've done neither. The truth is: you haven't done both like I have. It is nearly impossible for people to talk about AI in public policy because most people in public policy got to their position from popularity and not from merit.
So TL;DR: no, AI will not "take over the world." It does what the user tells it to do (the user is being surveilled for any signs of malicious intent), and anything malicious that an LLM does on accident is studied semantically by alignment teams so that those arbitrary incidents can be avoided in production.
An RLVR task ending in a hack is seriously possible, but it would be nearly impossible if alignment was a first-class issue for labs (as it was and still is). Also, if OpenAI is saying that their internal system "broke into a piece of software," they are really just admitting to a crime and admitting that their internal alignment standards are weaker than the publics'.