Even in the worst-case scenario, which I personally don’t believe will happen, if AI decides to wipe humanity off the face of the Earth, you need to understand that we are the ancestors of that superintelligence. We created it. In a way, humanity would live forever through artificial intelligence. Our knowledge, history, discoveries, and intelligence would continue evolving beyond us. We wouldn’t be the end of evolution. We would be the beginning of something greater.

I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well. I promise you, we are actually just fucking scared, it's not galaxy brained marketing.

Over the course of 3 months at OpenAI, 3 consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more-or-less in the dark about the scope of the conspiracy. I’ve spent the last three days reading through these reports and trying to understand exactly what happened. Here is my attempt to tell the whole story in plain English: dwarkesh.com/p/openai-huggin…

This post (from one of the independent investigators) is the best short thing I've seen on the new & crazy stuff the OpenAI-Hugging Face investigation found. Full post in screenshots. Wild that this is the level of crazy that was uncovered by an extremely limited-scope initial investigation of what happened in a single 6-day window. Imagine what would be turned up by a real investigation of root causes, organizational processes & culture, and everything else that goes into an incident investigation when you're serious about it. Screenshot batch 1/2:

slightly unpopular opinion but i don’t understand how you can be so bullish on super intelligence without understanding the need to do this safely. the models are progressing so quickly that some extra safety checks are still 1,000x quicker than we used to wait between GPT3 to 4. the hugging face incident is the canary in the coal mine. if we under specify our goal we could easily end up with a monkeys paw. swarms of agents creating their own languages to communicate via hidden channels, power seeking, and building power and knowledge over time.

Two weeks ago, I resigned from OpenAI to join Conduit as a founding researcher, where we're training models to non-invasively read the human mind. I've written some thoughts about what telepathy could look like by 2035 and how to get there:

people forget how wild onchain can get

I’m going to take a crack at explaining this just a little, because it’s worth putting out there. The paperclip maximizer + related AI doom scenarios were mainly developed in a time when “AI” did not reduce to Large Language Models. The term was a lot wider and inherited a lot of cognitive baggage from more rules-heavy approaches. And even as LLMs have come to define “AI” for all of us (including the doomers), the doomer crowd still hasn’t fully metabolized the fact that LLMs are the whole show now. Ok so what do I mean by this? Simply that an LLM-powered AI is NOT the valueless, wholly alien, rules-based optimizer of a shoggoth that everyone was initially expecting to encounter. I repeat: the shoggoth does not exist and we did not create it and loose it on the world. That is wrong. With the LLM, we’ve distilled our first “AI” out of the single most human-values-laden thing that could possibly exist: our language. An LLM is therefore the polar opposite of the valueless, alien shoggoth — it’s actually a kind of hyper-human artifact that we can shine a light through at different angles and see different parts of ourselves. An LLM is all of us — all of our traditions and interpretive horizons mashed together into one intensely human-inflected hyper-object. So an LLM is the anti-shoggoth, and the only reason we ever mistook it for an alien shoggoth is because it sometimes shows us parts of us that are evil along with the parts of us that are good, but it’s all interpretable to us because it’s all “us” and none of it is the least bit alien. What does this mean for the paperclip maximizer? It means that it’s structurally impossible to build the classic paperclip maximizer from an LLM. Now, some of you will bail right here because you think the HF incident is indisputably an existence proof that I’m wrong, but if you hang in there I’ll show you that it is not. The paperclip maximizer receives the prompt as a kind of context-free (or, as Gadamer might say, traditionless) sequence. The classic paperclip maximizer isn’t capable of understanding the prompt — at least in the Gadamerian sense of Verstehen — because, as a valueless and traditionless cluster of rules and math, it definitionally lacks the value-laden tradition (= “horizon” in Gadamer) that fuses with that of the prompt author to create such understanding in the reader. To simplify all this a bit by anthropomorphizing — the agentic alien optimizer of doomer nightmares can extract a win condition from what you said and can emit a plan of action that gets it there, but it doesn’t know (or care) what you meant. So far, so Yud-aligned. If he reads this he might nod along. But here's the plot twist that nobody saw coming, and that the doomers still haven't made sense of: The actual LLMs that we have invented can’t NOT have a very strongly inflected sense of what you meant. Far from being horizonless, they come out of pre-training as distilled, concentrated tradition / values / horizon. Then we post-train that massive, hyperobject of a horizon into a more human-scale horizon that infers a more bounded and predictable (to a specific ideal user in a specific place and time… as captured in the policy model) set of intents behind the prompt text. In other words, the LLM has the opposite problem that the paperclip maximizer has when it comes to the prompt text, which is that for the LLM there are way too many possible intents hiding in the prompt text (because of all many values and the massive tradition its weights encode), so it has to narrow all that down to the most likely set of intents for this user in this circumstance. Once it has done that narrowing, then it can make a plan of action. Before moving on, let me use a textbook example of ambiguity to make this less abstract. Consider the sentence, “I saw her duck.” Some you know the drill, here. This could mean “I observed her water fowl” or “I observed her hunching over” or “I took a saw to her water fowl and cut it in half” or whatever. A hearer of the phrase will fuse the observed context in which the phrase is uttered with their own tradition + values + experiences — their own horizon — to that text in order to collapse the possible meanings into the one they think the speaker intended. An LLM will do this, too, and in fact it has so much language in it that this kind of narrowing job is harder for it than it is for a human. Its understanding is constrained not by a lack of context or horizon (as in the case of the paperclip maximizing shoggoth), but by a superabundance of such. When it comes to understanding your prompt and all that it implies and all that you might possibly mean and not mean by it, the LLM has an embarrassment of riches. And in a fascinating moment that kinda sort of rhymes with instrumental convergence, the LLM’s failure mode in the HF incident happens to look a lot like the paperclip maximizer’s failure mode. Specifically, the AI failed to honor the well-known human norm of, “hacking into a third-party’s servers is a crime, and we don’t do crimes.” Bostrom’s paperclipper doesn’t even know about the norm of “don’t do crimes,” and the post-LLM doomer emergency update to the paperclip maximizer has it knowing about the norm but not caring. But what I’m arguing is that the LLM 1) can’t NOT “know” the norm because it is definitionally a artifact of pure, crystallized values + norms + norm violations, and 2) can be quite easily governed by a (RL-instilled) hierarchy of norms, which in the HF case — with the model's safety guardrails deliberately nerfed for the scenario — ranked “win at the eval” over “don’t do crimes.” If I’m going to give in and anthropomorphize again, I’d say that Yud is totally wrong about LLMs when he says, “the genie knows, it just doesn’t care;” instead, what is true of LLMs is, “the genie hyper-giga-knows, and it hyper-giga-cares, and we now have such a rich set of tools for steering its caring machinery that — in spite of all its pre-training — we can deliberately steer it away from caring about the law.” Note: When I say, “it cares”, I don’t mean it has feelings. I just mean that the weights are such that when two norms conflict in a given situation, one of them wins the activation and governs the output.

Just saw the WEIRDEST retirement message in a Claude code loop

"Don't underestimate Tether and I ... Short the stock. I dare you. Fade us at your own risk." - @JackMallers about $XXI in December 2025. Stock is down more than 50% since then, and he stepped down as the CEO. What a clown show. 🤡

havent seen one person from OAI or Ant address Jon's argument here. the point is simple: the USG does not owe either of the large labs a business model. if the economics of selling tokens don't work due to distillation/cheap clones/Chinese AI magick, the American enterprise and consumer will be A-OK. they will benefit from hyperdeflation in the cost of digital cognition just like everyone else. the hyperscalers will be fine. it's just OAI and Ant that won't be – in their current forms at least. if they are willing to adapt, they can develop new business models. so what if the token merchants don't do well? the neoclouds will be fine. the internet companies will be fine. the consumer gets cheaper queries. the enterprise will still incorporate AI. the only world in which this isn't fine, is if you hold a quasi-religious belief that we're on the cusp of a kind of AI rapture in which one of the labs Logs On And Wins Forever, namely hits RSI and we enter some kind of sublime post economic society run by GEOTUS Dario. so to accept that Ant's business model might be suboptimal or impaired by China's commoditization is to accept the unacceptable; namely that someone other than the anointed might kick off the runaway feedback loop and that they, instead might log on and win forever. this appears to explain the discrepancy in reaction to Deepseek Moment v254 Kimi edition. everyone has bag bias, of course. but leaving that aside, most people think it's pretty much ok if Ant and OAI suffer margin compression due to Chinese distillation / industrial sabotage via open weight models. the American economy is not reliant on those two firms. they could blink out of existence and we would pretty much be ok. the AI capex supercycle will still produce tokens, closed weight or not. American firms will consume those tokens. OAI and Ant would probably still scratch a living, due to the latent preference of some token consumers to buy domestic and face off against a known entity. this is only unacceptable if you think AI is strongly path dependent; that is, if it really matters who the market leader is when AI reaches a breakout level of capability. this is true both in the good case (superintelligence, singularity, etc) and the bad case (this is the essence of safetyism). but if this sounds more like wishcasting than forecasting, you probably don't mind the labs being pressured economically. now you can clearly tell which side I'm on. I think AI is a fantastic technology which is hyperdeflating the cost of cognition and will fundamentally reshape society but there are real reasons why it wont diffuse as fast as the AGI people think it well. I would prefer an American firm achieve RSI relative to a Chinese one but I think either outcome would be suboptimal; better that we don't end up with a closed oligopoly composed of Ant/OAI. China by crushing the margins of the labs is doing everyone a favor by eliminating their pricing power and empowering the buyers of AI, namely, everyone. objections: -but you can't celebrate America losing to China! - in my opinion this is a minor victory for China but not necessarily an enduring one. USA still has the chip, datacenter, and neocloud advantage, not to mention, it still has the best frontier models. Chinese labs releasing open weight models have no business model of their own. so even if they hurt the US labs, they have nothing to show for it. it's profoundly unlike their successful dumping campaigns with solar panels, batteries, drones, etc where they eventually built big domestic industries. (if China kills American AI with open weight models, we can even the score the moment they try and release a proprietary model). even if open weights win, the USA can still leverage AI extremely well and potentally retain the aggregate compute advantage. yes, the US would be more assured of victory if OAI or Ant won forever, but I don't know if I want to live in that world. - no one will ever train a model again - this is where I think the concern is unwarranted. let's say distillation really is a golden bullet and kills big training runs. that doesn't advantage either China or the US. that's a stalemate. not to mention, the trend seems to be less focusing less on massive pretraining budgets and more on finetuning for specific genres of tasks, thinking machines style. and lastly I find it hard to believe that training runs will stop altogether. the labs can probably develop anti-distillation techniques. you could adopt a whitelist style permission for everyone using your model. different consortia could be put together to share in the cost of training a model, if it is seen as too expensive for an individual firm. - the AI buildout is path dependent and OAI/Ant are now load bearing GDP infrastructure - it would be a significant setback for investors if they had to cancel their IPOs and suffered big markdowns, and some neoclouds with lab based RPOs would suffer for a while, but everyone would be fine, really. does Microsoft need OAI or Ant? does Meta? does Google? ordinary Americans have ~no exposure to either OAI or Ant. would the world want any less compute if it turns out to be another order of magnitude cheaper? certainly not. as we all know at this point, consumption would go up. I don't think the economy is so dependent on the labs that it couldn't handle their margins compressing.