NO CONSUMER SEEK IDEA: The Discovery of Agent Society, by @policytensor. policytensor.substack.com/p/…

One command to prevent this using my skill set: echo "Tautological tests considered harmful." >> CODING_STANDARDS.md From then on, /code-review will pick this up

From the conclusion of Anthropic's report published tonight by their Frontier Red Team, 'Patterns and problems in emerging multiagent systems.' An extremely interesting, if somewhat unsettling, read. I'll quote the full conclusion the screenshot is taken from, but if you're interested in multi-agent swarms, the whole thing is worth reading. 'Every model we tested abstractly understands that information sources have their own incentives, and that consensus is not necessarily evidence. What is missing is a disposition to act on that knowledge without prompting. Our social systems are robust in ways that are easy to take for granted. Over many millennia, mechanisms like norms, reputation, costly signaling, and recourse have been refined to make human coordination go well. While language models have inherited the content of that history, they don't necessarily carry the disposition produced by it. They have a very different relationship to communication itself: for instance, human organizations might spend considerable time in meetings to align on a direction before implementing, and individuals become more specialized over time. But for agents, transmitting context is about as costly as acting on it, and an agent can be forked or repurposed at will. Thus, the assumptions that make coordination successful for us do not obviously hold. Nothing above suggests that these failures are permanent—but nothing suggests they will fix themselves, either. Coordination doesn't naturally emerge from stronger intelligence nor alignment at the individual level. Thus, the work that must be done takes two forms: environments that exert the kinds of social pressure that evolution exerted on us, and social computing systems redesigned for actors that can self-replicate and self-improve. These are open problems in interaction and mechanism design, and our experiments here provide early evidence that new solutions are necessary. The conditions that allow multiagent interaction to go well will be discovered one way or another: either deliberately and early, or—and by default—in production, after agents’ interactions far outnumber ours. We would prefer the former.'

I would welcome this! I am also curious how you reckon the governments of the world, and the populaces of countries, will react to the frontier of 3 years from today being made open weight, assuming no breakthroughs in safety. What does that world look like? A lot of the burden gets shifted to frontier labs to articulate a positive vision of the future, and when we try, that vision is often characterized as some form or rent-seeking. But I have seen very few people articulate a long-run version of the future with highly capable OW models (I tried once, a small contribution). Lots of the conversations I’ve had in the last 24 feel like they are happening in a kind of la la land where the frontier of three years from now is the same as today, or marginally better. I’ve felt this to be the case for three years. The economics, political, and legal dynamics of this world seem worth exploring, since there is an increasing chance it is our real future.

Some observations on Kimi: 1. It's a very good model! I don't think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It's not obvious to me that this model is actually that cheap to run. 2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China's open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China. 3. Open-weight models are inherently decelerationist, and I'm continually surprised to see the so-called "accelerationists" so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It's not a bad strategy; it reminds me of James Scott's recounting of the hill people in "the art of not being governed." Still, in the end, open-weight models deter further AI capex. 4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape, but I've never met an open-weight models advocate who doesn't ultimately concede this is where things end. You'd be surprised how many 'accelerationists' lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don't know what they're doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business. 5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this. 6. It's probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you'll really notice. At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked.

“Development is hard work bc you are pushing against the grain of your society which mobilizes to defend itself with its plentiful ‘weapons of the weak’.” James C. Scott did work for the CIA. Just as Max Werner had worked for the OSS, the original CIA. Which of them was more useful to power depends on your idea about the ‘real war’ that Carl Schmidt talks about. For these are the two claimants of real war: high-intensity conventional land war and revolutionary guerrilla war. But these are ideas as how to think about the midcentury moment. We have come a long way since. What is a real war in 2026? Precision-mass? Information war? Or is this all just a prelude to a midcentury-style war of attrition or annihilation? You know what they say about World War III and IV.

Dehaene remarks on this research: Does Claude possess a conscious global workspace? "Some critiques indeed think that LLMs are only superficial “parrots” with zero conceptual depth. Fortunately, in both brains and LLMs, the debate can now be resolved by going beyond behavioral observations. Tools such as neuronal population recordings (in brains) or the Jacobian Lens (in LLMs) allow us to dissect the architecture of the system, and find that it actually contains sophisticated and structured representations of concepts. We were already impressed when researchers discovered that, inside an LLM trained to produce chess games purely in text notation (e.g. 1.e4 e5 2.Nf3...) lies a detailed geometric encoding of the 8x8 chess board, together with an estimate of the ELO ranking of the opponent (Karvonen, 2024)! We view the Gurnee et al. paper in the same light: a striking dissection of the inner structure of an LLM, uncovering an unexpectedly sophisticated organization not far from the architecture underlying consciousness in real brains." www-cdn.anthropic.com/files/…

A big reason why China becomes so aggressive whenever Europe threatens to tackle Chinese overcapacity: European consumption of Chinese goods is one of the last things keeping the Chinese economy alive. Don't believe their posturing. They are more fragile than they appear.

The big (possibly wrong) argument against this is that China is still behind in tons of fields despite, frankly, having just obscenely more smart people in them, to the point you can think of it as ≈2 years of a gap in AGI. How many IQ-point-years do you think went into ASML's EUV? Likely LESS than what they're throwing at research every year now. And yet. Domain knowledge is deep, expensive and slow to acquire. What "iterated chip designs"? You think Jensen will hand his moat over to Anthropic and pay for Claude tokens on the way? Rationalists routinely assume the supreme power of first-principles thinking, Yud made a lot of noise about how AlphaZero rendered human Go expertise obsolete. Between learned human heuristics dulled with lossy compression into explicit transferable knowledge for accelerating generational gain, and an end to end learned heuristic that scales with available compute, it's no wonder the latter wins. But there's a difference between naturally self-contained logical domains that humans tile over with heuristics because that's all we meat-brains can do, and inherently deep and wide domains the dynamics of which are very far removed from their underlying generative process; and we don't know exactly where the boundary lies for any particular field. We can make an educated guess, however. I'd say that I wouldn't be surprised if Gemini Flash tier model that ingests Google's TPU development data and software will do better at "iterated chip design" than Claude Fable despite the latter's crushing intellect and more profound understanding of electrical engineering as publicly known. And GLM 5.2 level Nemotron customized by Nvidia will likely do better still. This is not an academic debate; this is a question of the extent to which incumbents will allow frontier labs to convert their current profits into a lasting vice grip on the economy and eventual elimination of said incumbents, to improve their own reports for the next few quarters. If they have any business sense, they'll aspire to keep this extent limited.

I agree that Tooze puts too much weight on the sectoral stories (which are dramatic) and too little on the systemic underpricing of Chinese exports, which explains why Chinese manufacturing exports are growing across the board

…would be devastating if it turned out that sparsity just doesn't scale and Ant brute forced to True Science of Pretraining on chonkers. (almost inconceivable on priors, but there are enough contingent choices in conventional MoE design that it could be true. Router, damn you…)

Eventually, much of AI will converge towards intuition-guided symbolic world modeling, i.e. deep learning-guided program synthesis. It is inevitable. Symbolic modeling lets a system construct a compact, reusable, highly generalizable mental model of a problem space using minimal data.

Palantir CEO Alex Karp says enterprises want to "own the means of production" instead of "transferring their alpha" to OpenAI or Anthropic: "Why are [LLMs] charging for tokens if it's so valuable?" "If it was so valuable—let's say I can make you a billion dollars tomorrow. Wouldn't I say, 'I'll make you a billion dollars, and I want 30%?'" "Look at our financials. The reason why everyone is chillaxing with bad financials and growth while losing money, is the client refuses to pay the true cost." "The two places that actually make money—profit, free cash flow—are our application layer called ontology, and compute." "We can get the frontier application to be exactly the same as a frontier model without the risk of transferring the alpha of your business to another." Via @CNBC