They said exactly the same thing 2 years ago. Nothing happened and we’re ok.

hmm there are reasons to consider a pause but also some subtext here worth considering.

No!!!!!! Stop the lies. You have an agenda. You dislike AI for YOU. Stop pushing your fears on others. And Anthropic can suck eggs. •

This is an #OppenheimerMoment. The brakes will only come if there is some trust. But how will there be trust between the US & China? The SALT arms control agreement happened during the Nixon/Kissinger era of detente & faded under Reagan. The first START was signed as the Soviet Union was weakening, right before the August attempted coup that preceded the denouement of the entire Soviet empire. And as we know now, the latest START just expired. This means arms control, whether nuclear or AI, really only happens when broader tensions relax for whatever reason. Thus, the prospects are not great that this will happen at present no matter the rhetoric or diplomatic leaves that may be offered.

Fear mongering as marketing positioning is rich

How about players building unstable, chaotic, unsafe, hallucinating, and untestable black box systems with unpredictable intentions STOP doing what they are doing. We CAN build all that humanity needs to cure disease and build many technical marvels WITHOUT these unpredictable monstrosities. The problem is not computing intelligence broadly. The problem is the particular form/architecture and specific designs that a few companies are inflicting on the world.

If a team of AI developers says 'recursive' or 'swarm' and their code has no timechain equivalent they don't understand AI systems at all and are either fooled by the rearranging of text or genuinely scamming.

@AnthropicAI made it up and you fell for it

You must learn to distinguish between marketing hype and reality. The article is bait-and-switch. To draw you in it gestures to hypothetical future AI that doesn't have the *fundamental limits* of what's currently called AI. Then proceeds to talk about advances in coding.

i get why a pause sounds sensible. i just don’t know how you make everyone believe the other guy actually paused too. that’s the bit where “slow down” starts looking like another arms race.

Self improvement is only possible if AI will produce better results it could learn from. There is a valid argument that AI may actually produce mediocre results and by learning from it will reduce quality rather than improving it. We don’t know the answer yet and it could be different in different fields of human knowledge.

This is extremely concerning when you consider what it would take. At the very least a global government with total visibility on datacentre traffic, and the means to punish those that cheat.

1. Isn’t this equivalent to John D. Rockefeller asking to slow down pipeline construction just when he achieved dominance. Who benefits from the slow down? Isn’t bureaucratic interference worse than the alleged danger? 2. What happens if China does not slow down, or claims to slow down and doesn’t?

Give me a break, do you think the Chinese care about safety? There will be no global effort at AI regulation because the incentives to cheat are just too high.

I think an important related issue is the RSI makes the whole AI race much less predictable. State actors that are behind (or perceive themselves to be) might choose to act once RSI looks imminent—or they perceive it to be imminent.

This is unworkable because cheating can't be detected as it mostly could in the nuclear sphere. A regulatorium just hands AI to the bad guys. Recognizing the dangers in AI means actively using AI as a whitehat in what will be a never ending battle against the blackhats.

Why do we think it's even possible to have a concerted global pause on development. China is only 6 months behind the frontier, if that. China has centrally located resources to give Chinese labs the best chance to catch up and overcome lack of chips and resources compared to US frontier labs. If Western labs pause then China just get there first which is a much worse thing than Anthropic getting there first. I think Dario et al have some good intentions by positing this but it's almost like setting up a "We told you so" for later. "We wanted to pause last year but the Chinese/OpenAI/X kept chasing us, so we had to continue." "We didn't want to release Mythos so widely but we had to in order to improve our revenue so we could raise enough money from our IPO."

I haven't read the article , but does it mention some AI Treaty between US, China, Russia, etc.,?

The global adult in the room,China, Has a plan of course Wouldn't you know it While the West kicks around Woolly bullshit ideas That will never be looked at

“Who are these angels that will organize society for us” and constrain the AI race in a rational, effective, and efficient manner? Note: The quote is from Milton Friedman.

The arms control analogy is right but it's also the problem. Nuclear verification worked because missile silos are visible. Training runs aren't. The incentive to defect quietly is enormous, whoever pauses while others continue loses the race permanently. Coordination regimes without enforcement mechanisms are just statements of intent.

The paper is actually heavily caveated. Anthropic explicitly admits in the fine print: "It is not yet clear that Claude is capable of research judgment" As long as human judgment & direction-setting are required, AI is not recursively improving itself

Anthropic continues to use this formula to fear-monger and then fundraise . It has been working for them and will continue to do so.

Marketing efforts from Anthropic

I think the horse has already bolted. By the time any international agreement is hammered out we'll have super intelligence anyway. It's likely coming in the next few years.

Every lab fears being left behind more than it fears a joint slowdown. That’s why AI isn’t really an arms race between nations—it’s increasingly an arms race between a handful of well-funded companies, each warning about the dangers of acceleration while accelerating. Fear the race. Lead the race. Regulate the race. The modern AI playbook appears to be attempting all three simultaneously.

A far better alternative is to master AI before the wrong people do. AI is a productivity tool. We need to exploit it before the bad guys do.

It’s inconceivable at this point that there would be a voluntary slowdown. Maybe lip service toward such a thing in public. But behind closed doors in the US and China, it will be full steam ahead with even more resources. Potentially the most powerful tool in history awaits the winner. Mere humans, with their avarice and lust for power, will never be able to resist.

How embedded in AI research are you Niall? Do you - not use it at all-occasionally or push the LLM until it unconfuses Confucius ?

Curious...

You do not pause software development due to security issues. You develop defenses. Look at the history of software. All of these people are lying. They have a hidden agenda.

Delaying AI development looks like an impossible task, unfortunately

Yes, exactly. Given the pace of improved capability & welk-stated risks, the race logic ( is there a finish line for tech?) makes little sense for humans not building it. The Pope has a point.