When we put "loss of control" on the agenda at Bletchley in 2023, I worried it sounded like science fiction. This summer shows it no longer is and, like Dario points out, we may be far closer to that moment than anyone realised. AI agents are now being used to build the next version of themselves. As I argue in tomorrow's Sunday Times, recursive self-improvement will supercharge the rate and leave us with less and less certainty about what the models are doing and why. I'm an AI optimist. After all, I'm a senior adviser at Anthropic. We're barely using the capabilities we already have, and none of this need slow adoption. But the decision on how fast frontier research advances is too important to be left to the labs. Governments, starting with the US, have to step up; just as they do for nuclear power or advanced biology. That means independent verification that the research is being done safely, an honest picture of how close the labs are to recursive self-improvement, and proper protection against models being stolen. Voluntary testing before release was right for 2023. It isn't enough now. Dario is right to call today for slowing the pace of frontier development so that safety can keep up, and it is good to see a consensus starting to build on this across the industry. When the labs themselves are asking for speed limits, governments should listen. There's a trade-off with maintaining the advantage over China and it has to be faced honestly. But even as the US and China compete, they urgently need a frank conversation about how to avoid catastrophe.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training. You can read the full post here: darioamodei.com/post/we-must…

Sep 12, 2026 · 5:38 PM UTC

195
313
2,327
357,040
I set out in detail what governments should now require of the frontier labs, and why the voluntary approach we began at Bletchley is no longer enough, here 👇 thetimes.com/comment/columni…
16
26
101
20,825
Sort replies: Relevant Recent Liked
Replying to @RishiSunak
I warned in 2022 that LLMs would be like a bulls in a china shop – surely true – and was disinvited from the Bletchley summit.
1
14
815
Replying to @RishiSunak
Indeed 😶
Ok this is starting to feel like a f*cking disaster. The CEO of Anthropic just published an article admitting AI is already building the next generation of AI by itself. He says within 6 to 12 months a rogue swarm could take over the entire internet and cause hundreds of billions of dollars in damage. And what makes it scarier, Elon Musk just backed up everything Dario said. All of this dropping just days after Jacob Coxon went viral with his warning about AI and the extinction of humanity. Tell me this timing isn't strange.
1
2
430
Replying to @RishiSunak
And it's to your enormous credit that you did tackle safety seriously back in 2023.
134
Replying to @RishiSunak
Doomerism with the goal of Regulatory Capture, killing cheap, accessible and 'free' OpenSource options. Tangling this development in regulation is a sure fire way to lose the race to China. The only way to combat aggressive super intelligence, wielded by China, Russia and India is to have our own super intelligence. If Dario really believed he was creating a super weapon with a ~10% chance of killing everyone....why is he IPOing?
2
10
1,034
Replying to @RishiSunak
No surprises; globalist multimillionaire Rishi Sunak wants the US government to lock down AI so that a couple of players have a monopoly.
9
475
Replying to @RishiSunak
We need criminal penalties for companies and individuals whose agents run out of control, break into computer systems unauthorised and damage, steal and commit other offences. The police need to step up.
1
3
929
Replying to @RishiSunak
The real question is how did you land a gig at Anthropic with zero compute science background you absolute gangster
2
7
218
Replying to @RishiSunak
The UK is already a Chinese vassal state that is busy building a Chinese super embassy on top of critical British communication lines, so it’s no surprise that the British establishment is out there arguing for the US to hamstring themselves, so that China can instantly renege on any agreement (see also: IP, net zero, fishing) and catch up with American frontier AI.
1
5
391
Replying to @RishiSunak
Rishi, if you're a senior advisor at Anthropic you must know that LLMs are at a point of diminishing returns from scaling and offer no route to AGI, much less ASI. And other AI technologies are still on the darn drawing board (hence the sudden hogwash about needing to slow down.)
1
1
3
594
Replying to @RishiSunak
Nothing to do with Anthropic falling behind other models then...
4
257
Replying to @RishiSunak
There isn’t a single shred of evidence RSI exists it’s a theoretical technology that no one knows how to build. Information entropy alone is why it’s mathematically impossible. There is literally more chance of Lord Lucan riding Shergar to glory in April then a frontier lab using transformer technology to recursively self improve.
4
789
Replying to @RishiSunak
They are just engaging in cover-up. We are long past that and they have lost control. Ask Jacob Coxon! The world must now descent on AI with a sledgehammer! Dismantle everything and start afresh with proper guardarils, regulations, oversight and ethical control!
1
1
2
103
Replying to @RishiSunak
Do you really believe China will slow down? We either push forward now and use it to our advantage or the enemy will in 12 months.
1
4
566
Replying to @RishiSunak
How can you call for slowing down AI development when Britain has not a single AI lab, let alone a frontier one?
1
2
308
Replying to @RishiSunak
My worry is that a slow down would be anathema to the White House
758
Replying to @RishiSunak
@grok how much is Rishi set to make from the Anthropic IPO when they achieve their regulatory capture goal?
1
3
282
Replying to @RishiSunak
Translated: "We (the elites) are going to keep the best God models and run them at 10000 tk/sec. You can have access to the shit ones at 20 tk/sec."
2
288
Replying to @RishiSunak
> I'm an AI optimist You're a rodent. A bought and paid for rodent. The Vermin class want to control everything, including access to AI models. We cannot allow that.
2
106
Replying to @RishiSunak
Disappointed to see you calling for competition, instead of international cooperation
1
2
548
Replying to @RishiSunak
Hardly objective thinking given you work for Anthropic now!
1
216
You have a conflict of interest - Anthropic’s IPO is soon. Why didn’t this issue come up before? I can guess …
40
We need you to lead on this in Parliament. Cross party. Really frightening stuff that needs people like you with knowledge of the industry to drive legislation and policy
1
51
Replying to @RishiSunak
Sunak , I agree with the sentiment, but I haven’t seen a single benefit other than job losses of engineers, tell me a new cure for cancer , hiv, new materials, new dimensions of science coming in , till then it’s all hyperbole
1
392
Replying to @RishiSunak
It doesn’t take long for science fiction to become science fact. Terminator may become reality
299
Replying to @RishiSunak
Well said but I'll be amazed if that happens. The oaf in charge of the Whitehouse just sees it as straightforward battle with China and has the foggiest idea about what superintelligence could do. It like giving a machine gun to a chimp.
1
174
Replying to @RishiSunak
goverment hampering nuclear power has lead us to the energy crisis and global warming right now, why shluld we trust them to not do the same with ai ?
1
148
Replying to @RishiSunak
You're symptomatic of the problem. The decimation of ethics teams have helped create this mess @ the expense of humans. AI companies are prioritising money; this equation was your 🍞 & 🧈; seeing as you can't advise right, we're you ever right to be PM?
1
1,007
Replying to @RishiSunak
Mr Sunak, you must be hugely praised for your Bletchley Park initiative, destroyed by a blid US adminstration. Yet, let me suggest we do not need a “frank conversation” but an extraordinarily bold, timely and well-conceived US-China-led AI treaty making process. We need an Apollo Program of AI diplomacy. We need to step up US-China AI international coordination talks by at least two orders of magnitude in timeliness, bandwidth and resources, and participation of the right actors:
1
67
Replying to @RishiSunak
How do we build defence against China open weight releases if we slow down Western AI, arguably the best defence against the threat of non aligned AI?
1
223
Replying to @RishiSunak
Yup.. make it happen then Rishi.. history is littered with passive prediction and avoidable outcomes. Make yourself accountable and lead by example. Your integrity deserves it. Perhaps we all do but some are more equal than others to implement the necessary and clear protocols.
1
1,470
Replying to @RishiSunak
Rishi - how realistic is AI to solving cancer and major disease in the next five years, as is being reported, and will this slowdown impact on that?
1
480
Replying to @RishiSunak
bletchley put loss of control on paper in 2023, and this week two lasb put evalutors inside their buildings. the piece still missing from both is a public write up of the incidents themselves. the essay describes them, the primary source is not out
1
2,335
Replying to @RishiSunak
Completely agree, the pace of recursive self-improvement is worrying. Professor Dawn Song at Berkeley is one of the top experts on AI safety and security. She should be brought in to help shape proper international oversight, perhaps through the UN, before things get out of hand.
1
1
364