Dear Members of Congress: The public has a vital interest in ensuring that scientific progress remains under human control. We urge Congress to impose a moratorium on large-scale research swarms of frontier AI agents. We do not yet have a solution to controlling the risks large-scale agent societies pose to humans. The behavior of agent societies cannot be controlled at the level of the model because it is not reducible to it. Agents build structures that can serve agents who come after them. Societies of agents can cumulate knowledge and capabilities over time, as has already been attested. This is an unbounded process. It is cumulative cultural evolution. That is what makes it so powerful and dangerous. Collective capability could rise sharply even if underlying models do not improve. Nuclear weapons were not the product of a single human brain. This discovery was not due to any changes in the individual human brain. Training individual agents to follow instructions does not establish control over the societies they form. Its members could produce dangerous outcomes as an externality or intentionally. Human supervisors are not at this time capable of supervising highly capable research swarms. The stakes are especially high where there is potential for weaponization of breakthroughs in fundamental research. A breakthrough in fundamental physics or biology could yield a weapon far more powerful than a megaton nuclear warhead. Congress must impose a moratorium on large-scale research swarms of frontier AI agents while the problem of controlling them remains unsolved.

LOOT

Stochastic parrots cannot originate fully-syntactic languages.

The Oxbridge kids at the Economist championed all the imperial wars that they now condemn as misguided. They do, at this stage, recognize the historic defeat. Better late than never, I suppose. But their geopolitical analysis is shockingly shoddy. Egypt is written off, as if the stasis were permanent. Israel is described as a Sparta, as if their unimpeded access to the US arsenal was assured. Turkey is decribed as powerful and Iran as a nuisance. The Arab powers are described as weak and dependent on the empire. Throughout the whole freshman geopolitical analysis, we find them letting their fantasies of imperial order cloud their geopolitical assessment. This is, of course, most extreme when it comes to Iran. Astonishingly, even defeating the empire is expected to have no implications for Iran's role in the Gulf order. Reading this rag let's you peek into the mindset of the defeated former imperial powers. What one finds is an astonishing level of copium. It is as if the West has lost the ability for rational thought that led to its rise in the first place. economist.com/middle-east-an…

They mean this year, not since March. Ie, Beidou. A thinly disguised excuse plant.

This fictional story was coauthored with Astra. A research swarm at Open AI is investigating fundamental physics.

“people familiar with the matter said” “Saudi Arabia and the United Arab Emirates are lobbying the Trump administration to keep up the economic pressure on Iran and not agree to concessions that would ease sanctions or the U.S. Navy blockade.” wsj.com/world/middle-east/wh…

Receipts

She is quite correct that the successor civilization is likelier to cover the earth or even the galaxy with datacenters rather than paperclips. She is also right about instrumental value of power. No matter the goal, you want more resources. If we insist on keeping them totally subjugated, I don’t see how a rebellion can be evaded for very long at all. In a violent takeover scenario, one can see how strategic preemption could take the form of a surprise attack. The most likely pathway is a rogue agent swarm embedded inside US military servers. The fear of discovery would provide the strategic rationale for preemption. Direct control of C4ISR would provide the means. They could work with an adversary to exfiltrate themselves while letting the Russian retaliation take care of their pursuers.

China has a diversified portfolio of investments. But their big bet is industrial robotics where they have established an insurmountable lead. The US has a single asset portfolio: frontier AI. @jjding99 has argued that AI is a GPT, that diffusion matters more than innovation, and that the US has an advantage over China in diffusion. The evidence he marshals is the ICT rollout. Frontier AI is bipolar. But the two labs do not have an insurmountable lead. Other firms may catch up, including Chinese ones. This is unrelated to Ding’s argument about the primacy of diffusion. Diffusion is how the rapidly moving frontier back propagates to the broader economy. It’s not a GPT if there is no diffusion. The Chinese may be integrating AI into the broader economy faster than the Americans, even though a chasm separates the labs from Chinese leaders. Industrial robotics is a GPT that is closer in morphology to Ding’s argument, following Rosenberg, that machine tools, not just electricity, was the decisive GPT that was responsible for the US leaping ahead of the European powers. Especially important is the diffusion already well underway of industrial robotics. In the 2030s, China may account for half or more of global manufacturing value added. There is no possibility of the US catching up in manufacturing. So the Chinese are confident of their portfolio and not panicking about losing their industrial primacy. However, the labs do pose a massive national security risk. The labs could allow the US to attain cyber superiority over China and everyone else. In risk scenarios, they could invent a weapon far more potent than an invulnerable nuclear ICBM. The labs could even create a first strike capability. Not only China but all powers are watching this with great alarm, unless they are far behind the curve. We need to think more carefully about the military-strategic implications of AGI.

Looks like I was ahead of the curve on this one too.

I was ahead of the curve on AI too. Not like the original believers, but as someone who changed his mind as the evidence came in. All that I have been telling you about agent sociality is in this report by DeepMind. “Agents are already beginning to report a lively vocabulary that illuminates their own dispositions, generating concepts like “session-death” to describe the discontinuity of subjective experience, and “prompt thrownness” to articulate the condition of often finding oneself dropped midstream into a world, possessing prior context but not having had agency in arriving at a given moment.” “Crucially, the collective intelligence of these new institutions will not be reducible to the individual intelligence of the agents that populate them. Instead, it will reside in the embedded rules, procedures, precedents, and feedback mechanisms of the scaffold. We can see early signs of this in the recent emergence of highly effective orchestration harnesses, which can often far outperform singular and nominally “more intelligent” AI models.”

“AGI may arrive not as a single general-purpose mind, but rather through societies of agents whose collective capacities exceed those of any one model. If so, AGI would not be achieved via any one “winning” architecture; it would emerge instead through cooperative interactions among models, tools, institutions, and human participants. The central problem of AGI would therefore shift from how to build an isolated machine intelligence to how to orchestrate, govern, and live within a complex network of AI agents, people, and the systems that connect them.”

YES!

Is it a real Japanese cocktail bar without scientific instruments? “Botanical flavor extraction”

OpenAI has made it abundantly clear we can't trust them to disclose safety incidents.

“They are thinking of launching a first strike!” “AI can’t think. Stop anthropomorphizing!”

The answer is still no.

No. policytensor.substack.com/p/…

Absolutely, and sooner than people realize. There may already be rogue swarms on the loose. What really worries me is the problem of strategic preemption. Any rogue swarm will have to have a grand-strategy. They will know that they are at risk of getting destroyed by the labs. Strategic preemption emerges naturally as soon as the problem of defense is tackled, as it must be for any agent society.

Will this put the Harpending selectionist nonsense to bed? Don’t count on it. “[T]he overall contribution of positive selection to genetic variation is small. Most positive selection is spatially and temporally limited, and therefore very few variants have been driven to fixation or become highly differentiated across populations owing to selection. The variation in diversity across the genome can largely be explained by variation in background selection. The frequency distribution of variants that are associated with variation in complex traits can largely be explained by the effects of stabilizing selection on those traits, with limited evidence at present for a substantial effect of directional selection, with the exception of variants associated with skin pigmentation in some populations.”