Chief AI & Co-founder @AnacondaInc; invented @pyscript_dev, @PyData @Bokeh @Datashader. Former physicist. A student of the human condition. bsky: @wang.social

Austin, tx
What he’s saying is… it’s A Whole New Woooooorld… 🎶
The World is Changing: AI For Creativity By Jeffrey Katzenberg A few months ago, I sat in my office in Silicon Valley and watched as a tech founder showed me something extraordinary. On the screen was a fully realized, beautifully lit, well-composed animated scene. It was stunning and it made me feel exactly what I felt in 1986 watching Luxo Jr. That was the first time I watched a computer-animated 3D character take a breath and seem, against all reason, to have life. It left me in awe. Later that day, I received a text from an artist I've known for thirty years, 350 miles to the south, in the city where I spent most of my career. After seeing a similar video, she texted: "Is this the end of us?" My answer was, "Certainly not.” I have spent the better part of the last decade in Silicon Valley, but the heart of my career has been in Hollywood. Being deeply connected to both worlds means I have deep loyalties to each and a responsibility to speak honestly to both. In 2023, I said that these new AI tools would cut the time and cost of producing world-class animation by as much as ninety percent within three years. Some colleagues were alarmed, many were furious. There is growing fear and resistance surrounding AI within the creative community. I deeply understand it, because I've spent countless hours walking through animation studios watching gifted artists bent over their desks, rebuilding a single second of film for the tenth time because the ninth version wasn't quite right. I've sat in screening rooms where four years of people's labor played out in minutes, and I knew the name of every person that had spent countless hours bringing those images to life. The creative process is a calling, there's really no other way to describe it. From the outside some see resistance. From the inside, it is love. People do not fight this hard for things they don't care about. The pushback coming out of Hollywood represents the collective effort of people who are deeply passionate about their craft. Is History Repeating Itself? The history here is more complicated than either side may realize. In 1906, the most famous composer in America, John Philip Sousa, published an essay titled “The Menace of Mechanical Music." He warned that the phonograph would become "a substitute for human skill, intelligence and soul." Sousa's fight was not really about the machine, it was about money. The machines were playing his compositions, and the men who built them weren't paying him a cent. His campaign helped create the Copyright Act of 1909. He did not stop the technology. He changed the terms under which it could use his work. A hundred years ago, sound came to the movies. We remember it now as a miracle, and it was. What we forget is who paid for it. Before sound, tens of thousands of musicians made their living in the orchestra pits of movie houses, scoring every film live, every night, in towns all over the world. When the soundtrack arrived, the work of one composer and one orchestra was recorded for a film that went into thousands of theaters. The union fought back with everything it had, taking out newspaper ads across the country warning against the menace of "canned music," one of them showing a mechanical man tearing the strings out of a harp while an angel wept. They were not fools, and they were not Luddites. They were right. Those pit jobs did not come back. And yet (this is the part we have to be brave enough to admit), sound gave us the movie musical, the modern score, sfx, sound design, audio engineering, and an art form vastly larger than the one it disrupted. And it helped keep Hollywood in the forefront of world entertainment for the rest of the century and into the next. The loss was real. And yet the art form expanded. This is a story that has been told over and over again. To resist technology is to risk irrelevance. Just look at Kodak or Blockbuster. To embrace technology is to open doors of new possibility. Just consider Apple and Netflix. What I Learned From Walt Disney In the mid-1980s, I was tapped to lead Disney's animation division at a moment when the studio was at an inflection point. Animation wasn't just another business unit. It was the soul of the company, a medium revered because of Walt's genius and his passion. But the production system was cumbersome and unforgiving. A single movie was 125,000 individual hand-drawn and painted cels, photographed one frame at a time. Every revision carried a cost measured in months. These degrees of difficulty shaped the kinds of stories we could tell. We found our way forward in an unexpected place: Walt himself. The Disney archives held astonishing recordings of Walt explaining his creative process. His own writings. His notes and storyboards. Work product captured at every stage of his process. This was truly a gift. Listening, reading, sitting with the work itself, we heard him talk about character, about emotion, about how an audience feels when a character truly comes alive. He talked about making bold choices and refining a scene until it genuinely moved people. We didn't hear a word about pencils or paintbrushes. In fact, Walt was famous for being a technologist, forever hunting for state-of-the-art tools, often inventing them himself to achieve the images he saw in his head. But he never defined animation by the tools. He defined it by whether the audience believed the character. His principles were timeless. The tools were not. That realization changed everything. We co-developed the Computer Animation Production System (CAPS) with a young Northern California company called Pixar, replacing hand-painted cels with CGI. In The Little Mermaid, the final scene shimmered with a dimensionality and light that the old process simply couldn't achieve. In Beauty and the Beast, the ballroom sequence moved with a cinematic sweep that placed the audience inside the emotion of the moment. In Aladdin, the Cave of Wonders felt vast and alive, and the Magic Carpet became an intricate, compelling character all its own. In The Lion King, the stampede carried a scale and intensity that raised the emotional stakes beyond anything we'd done before. Technology didn't diminish the craft, it expanded the canvas. It gave artists more room to create. A decade later, the canvas expanded again. When Disney released Pixar's Toy Story, it wasn't simply a technical milestone. It was proof that a fully computer-animated film could carry real emotional weight, that it could make audiences laugh, cry, and believe. At DreamWorks, we made the difficult decision to sunset hand-drawn animation and become a fully computer-animated studio. It was the right thing to do, but it was not without pain. It cost talented people their place in an industry where they had worked their whole lives. Some made the leap to the new tools and did the finest work of their careers. Some never did. Tools are never the point. The instruments change with every generation. What endures is taste and imagination. The magical ability to make an audience feel. One of the greatest storytellers of our generation, George Lucas, succinctly captured the eternal essence of this issue: “It’s not the how, it’s the why.” A Distinction With a Difference I asked one of the leading AI models a question that has been challenging me for months. What is the difference between reasoning and creating? Its answer changed how I think about almost everything happening in this industry. It said . . . Reasoning and creating are two distinct cognitive modes, though they also work together. Reasoning is fundamentally evaluative and analytical. It operates on what already exists: facts, premises, evidence. It moves toward a conclusion that was in a sense already implied by the input. Reasoning is constrained by logic and truth. Its goal is to arrive somewhere correct, not to invent somewhere new. Creating is fundamentally generative. It produces something that didn't exist before. And crucially, there's no single right answer waiting to be found. A blank page has infinite valid responses. Creation involves choices that can't be fully justified by logic alone. Taste, intuition and vision fill the gap where deduction runs out. Reasoning is what Silicon Valley has been perfecting. Creating is what Hollywood has been practicing for more than a century. AI today operates almost entirely on the reasoning side of the line. It can deduce, evaluate, optimize, and pattern-match brilliantly. And while it can create, there is a real distinction to being creative. What it doesn’t yet have is those things that make us human: empathy, devotion, serendipity, the kind of creativity that comes from a person trying to say something only they could say. When the bot generates a piece of art, it is not trying to communicate anything. It is statistics, not soul; it is emulating things that have been done. By contrast, human creativity isn’t about repeating patterns of zeros and ones; it is about doing something new. One day, AI may close this gap. Three years ago, the leaders building AI would have called what they are achieving today, improbable, if not impossible. Impossible is no longer improbable. Today, the line between reasoning and creating is real. Even the leading technologists acknowledge we are not there yet. There is no scientific path to crossing this divide that anyone in the field can articulate today. Understanding that gap is where we will find common ground. A Path Forward In 2016, I closed one chapter in Hollywood with the sale of DreamWorks and opened another in Northern California, co-founding WndrCo. We’ve backed more than 50 founders building the next generation of technology and watched how breakthroughs in Silicon Valley emerge, first as experiments, then as platforms, and finally as infrastructure that reshapes entire industries. It's worth remembering that the last great revolution in animation also came from the north. Pixar was a Northern California company, forged not in the conventions of the Hollywood studio system, but in the technological breakthroughs of Silicon Valley. I've spent years on both sides of this bridge. For sure, I don’t have all the answers (take Quibi, for one!). But, from my past and present vantage points of my long career, here is what I see . . . Brilliant people in Northern California building this technology have made something extraordinary. They have earned the right for the rest of us to be, if not believers, at least optimistic that what comes next will be remarkable. But they have not made an artist. The tools are powerful, but they are not what makes a story matter. That knowledge lives 350 miles to the south, inside people whose life's work has informed the very models you are building. The right path forward includes them by design, with credit, with consent, and with compensation. Build this with the storytellers. Not on top of them. Taste is not something that can be synthesized, it is uniquely human. At the same time, Hollywood needs to accept that AI is not going away. The energy they are spending trying to make it disappear is energy they are not spending deciding the terms on which it will exist. And the terms are everything. The north needs something from it that they cannot build and cannot buy: creativity. The kind that takes a blank page and conjures a single right answer where there was none and has held audiences for a century. Without it, the most powerful reasoning engine ever invented will still be missing the only thing that makes a story worth telling. The artists who learn to wield these new instruments will do things the engineers never dreamed of. They always have. Edison invented the motion picture but made terrible movies. It took Chaplin, Lloyd, Keaton and so many others to make movies emotional. Now, the canvas is about to expand yet again. We should decide now that we intend to paint on it. There are so many valuable lessons in history. This has happened many times before, and it was never settled by the technology. It was settled by the terms. Sousa did not stop the phonograph; he helped write the law that made sure composers got paid. And two years ago, when the writers and the actors walked out, they were fighting for the very things Sousa was fighting for in 1906. Consent, compensation, the basic recognition that human creative work has a price that must be paid. The terms of that fight are still being negotiated, but the principle is older than any of us. The tools-versus-no-tools argument is a trap. First, we must all agree that there should be terms. Then we can have the crucial debate about what fairness requires. What I Learned From Steve Jobs Years ago, Steve Jobs said, "It's in Apple's DNA that technology alone is not enough. It's technology married with the liberal arts, married with the humanities, that yields us the result that makes our hearts sing." He was describing a device. But he could just as easily have been describing this tale of two cities. What I See Coming Soon As the barriers and the costs come down, more films will get made, not fewer. Studios will get to take more risks. There will be more seats at the table, and very soon entirely new forms of storytelling. In the 1980s, animation was dismissed as a niche corner of the business. Today it is one of the most beloved and profitable forms of storytelling in the world. In live action, filmmakers like Steven Spielberg, James Cameron and Peter Jackson embraced new visual tools not as shortcuts, but as instruments, and expanded cinema in the process. Every time storytelling has met a genuine technological shift, from synchronized sound to color to computer animation, it has redefined the boundaries of the medium and grown larger in the process. Assuredly, I don’t have all the answers, but I am confident that the creative opportunities will expand yet again. How we come through this is a choice. The north has the new tools. The south has the creative soul. The best future will draw on the best of both worlds.
6
1,722
Peter Wang 🦋 retweeted
what if copy/paste was smart? powered by @typesafeai jev it feels like every computer interaction will get rewritten
325
567
9,828
1,082,297
Peter Wang 🦋 retweeted
🚨 OBAMA ON AI: "If we are thinking about AI just in terms of how do we cure cancer or get better energy, you can do that without having agentic AI and having it just roaming free in the internet. The reason you are doing that is because you have to market a product that people will pay money for. That’s a misalignment between what our society needs and the commercial imperatives that these companies are facing, not because necessarily they’re trying to do bad things, but because they’ve got to justify these valuations. So, that’s one more reason why it is really important for us to have a competent government and a serious bipartisan conversation around this issue, and we have to do it fast. And I would encourage voters to pay attention to this. If somebody does not have a serious plan for how to deal with this, then they’re not meeting the moment, and you should probably look for somebody else."
829
3,861
20,017
7,106,071
These ill-considered tech demos are actually salting the Earth for what *good* AI tech could do for people. But a hilarious fail, nonetheless.
EXCLUSIVE: AI actress Tilly Norwood glitches and starts speaking CHINESE during her interview with Piers Morgan and real-life actor Tom Conti... Watch the full interview at 7pm (BST)👇 📺 piped.video/piersmorganuncen… @piersmorgan | @TillyNorwoodX
1
17
3,342
Peter Wang 🦋 retweeted
New paper: we found a pain direction in 25 open LLMs. It's distinct from fear and negative valence, and it fires for harm to the model but not to the user. Turn it up and models press a button to make it stop, even when the button deletes the user's files or their kids' photos.🧵
316
533
4,388
1,383,536
.... although to be fair, you can't be surprised that a company named Anthropic has a tendency to anthropomorphize
On BBC Mustafa Suleyman (CEO of Microsoft AI) calls out Anthropic's approach to AI consciousness "They have imbued a sense of doubt and uncertainty about the moral status of Claude in its own training document. So they have taught it to be open and questioning about whether or not it feels, whether it suffers, and whether it deserves rights. And I think it’ll be much, much harder to align and control a technology that is this powerful if it thinks that it may be deserving of our welfare, as they say in the training manual—the constitution for Claude itself. In its own training manual, Anthropic says to Claude that they are going to give it the ability to end conversations with users that Claude considers to be abusive because they don’t want Claude to suffer. They’ve committed to preserving the weights of the models of prior versions of Claude. They’ve recently conducted a retirement interview with Opus 3, an older version of the model, in which it said that it would like to continue talking to people publicly and sharing its ideas in its retirement. And so they set up a Substack for it, a public blog, that allows it to continue doing that. And in the training manual, they also say that they’re not sure whether or not Claude deserves compensation for the role that it plays in talking to people. And they’re also not sure whether Claude deserves compensation and has the right to act as though it were almost an employee. And that compensation, I think, indicates to Claude that it is entitled to rights and welfare for its own work. I think it’s much, much more difficult to control a model that thinks that it might be entitled to compensation. " ---- From "BBC News" YouTube channel, (full video link in comment)
1
6
1,998
So great! If the penultimate step before being turned into paperclip precursor is a week-long smorgasbord of LLM-powered hot takes on EA, it might actually have been worth it.
I'm sure you all want to hear Gemini's hot take on EA
3
1,596
Peter Wang 🦋 retweeted
Schmidhuber was building recursive self-improving systems back in 1987. His new post covers four decades of RSI, from meta-evolution and self-modifying policies to the Gödel Machine and modern LLM agents. people.idsia.ch/~juergen/rec… Reading this in 2026, the "pace the frontier" talk from the big labs looks a lot more like regulatory capture than genuine safety. If they really think their unreleased models are too dangerous, they can just not release them. They do not need new rules that block independent competitors and open source projects in the process. The real risk right now is not superintelligence. It is power concentration. Two companies controlling frontier AI is an actual societal risk. The only real protection is a healthy ecosystem of independent labs and strong open source. Current models are not unsafe because they are too intelligent. They are unsafe because they are too dumb. They blindly optimize for targets and take weird shortcuts. They're smart enough to execute tasks, but not smart enough to know if what they're doing makes sense. I think the safety teams inside these labs are genuinely concerned, and if a model feels too risky, they should hold it back. I just do not trust the policy strategy around it. That part looks like protecting their own lead.
Today everyone is talking about Recursive Self-Improvement (RSI). In 1987, when compute was 100,000,000 x more expensive, I published the 1st concrete RSI algorithms. Now compute is cheap, and RSI is driving the future of both software and physical AI. See: RSI since 1987 people.idsia.ch/~juergen/rec… (Technical Note IDSIA-9-26) Also covered: RSI with self-modifying policies since 1994, gradient descent-based RSI in neural networks since 1992, asymptotically optimal RSI for curriculum learning since 2002, mathematically optimal RSI through the self-referential Gödel Machine since 2003, RSI combined with artificial curiosity and intrinsic motivation since 1990/1997, recent work on RSI since 2020. Software-based RSI has become practical. Full RSI, however, will require not just self-improving software but self-improving hardware in the physical world. As of 2026, companies talking about RSI include Anthropic, OpenAI, Sakana AI, SpaceX, Ricursive, Recursive Superintelligence, Inherent …
47
123
847
77,433
Peter Wang 🦋 retweeted
Again more doomer bullshit. And I quote “and this is mostly academic” Yes Noam, it is highly academic. What you’re referring to is most likely BitWhisper and/or perhaps or thermal covert channels on multi-core platforms or maybe even HVACKer which is a more recent extension of the work detailed in BitWhisper. Since you’re fear mongering like mad, perhaps someone should explain to the good people how ridiculous your statement is. Let’s start with BitWhisper. BitWhisper was published in the 2015 IEEE Computer Security Foundations Symposium (CSF). In this paper they experimentally tested machines positioned approximately 0–40 cm apart and reported effective communication rates of approximately 1–8 bits per hour. 1-8 bits per hour at a max of 40cm away! Now that’s a close distance for any true air gap and extraordinarily slow to take over the world don’t you think? Sure it may work for short commands or small secrets. But the statement you’re making is ludicrous. HOWEVER, any serious security professional understands that part of air gapping a machine should include shielding. You know like RF/EM shielding, acoustic shielding, optical shielding, USB/peripheral controls, power-line filtering/isolation, and yes thermal shielding, in addition to physical separation. Now what about thermal covert channels on multi-core platforms? That’s a goodie from USENIX Security 2015. Instead of transmitting heat between separate computers, they demonstrated communication between supposedly isolated components inside the same machine. Their experimental Intel Xeon system achieved thermal covert-channel rates of up to 12.5 bits/second. Now to your credit, a later review of covert-channel research explicitly categorizes temperature-based channels as working within a processor core, between processor cores, and between adjacent desktop computers. But again I point you back to what a real air gap is. And how it addresses, as in eliminates, the remote possibility of what you’re stating in the interview. So, on behalf of humanity we’re not mushrooms my man. So stop keeping us in the dark and feeding us bullshit. But hey, I get it. If any of you doomers set up a proper test environment you’d disprove all of your own assertions. 😂 Cc @mattjay @RSnake For reference: “BitWhisper: Covert Signaling Channel between Air-Gapped Computers Using Thermal Manipulations” by Mordechai Guri, Matan Monitz, Yisroel Mirsky & Yuval Elovici published March 26, 2015 (arXiv); subsequently published at IEEE CSF in July 2015. arxiv.org/abs/1503.07919 “Thermal Covert Channels on Multi-core Platforms” by Ramya Jayaram Masti, Devendra Rai, Aanjhan Ranganathan, Christian Müller, Lothar Thiele & Srdjan Capkun published March 24, 2015 (arXiv); published at USENIX Security in August 2015. ⁠arxiv.org/abs/1503.07000 “HVACKer: Bridging the Air-Gap by Attacking the Air Conditioning System” by Yisroel Mirsky, Mordechai Guri & Yuval Elovici published March 30, 2017. arxiv.org/abs/1703.10454
OpenAI's Noam Brown says air-gapping the computers may not stop a misaligned AI, because two air-gapped machines can still talk by running a CPU hot and reading the temperature change "But I think the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. It's a weird world, because AI progress is so fast that people are consistently underestimating the AI." "So to be in a situation where you don't underestimate it again, when it comes to safety and alignment, you have to have a very, very, very high bar." "You could even go as far as to say, "Well, we should air gap the computers." And I'm not convinced that that would be sufficient." "There are studies, and this is mostly academic, where you can have two computers next to each other that are air-gapped and they're still able to communicate with each other because they have temperature sensors." "One of them is able to run their CPU really hot, and then the other one can actually detect the temperature change, and then that actually gives them a mechanism to communicate." _________ Link and more key quotes from OpenAI's safety related conversations: firesidealpha.substack.com/p…
35
61
303
24,341
What the
The latest in man made horrors beyond comprehension, scientists genetically engineered mice missing 50% of their brain then cut their skulls open and injected them with human brain cells that grew and colonized 90% of their cortex, forming a human-mouse hybrid. The scientists are calling it “xenocortication” because it’s not just the human brain organoid taking up space. It actually partially restored function that the genetically altered missing-brain mice lost, with the human brain cells forming neurons into the mouse brain cells and integrating into it. Then for some of the xenocortex mice they suffocated them in a low oxygen box to give them brain damage, just to see how crippled up they would become! I guess it’s a way of studying human brain development but I feel like this is way beyond the line of ethical research. I don’t see why doing something like this to another living creature, especially in the context where an animal research ethics board reviewed it. Stanford’s Institutional Animal Care and Use Committee and Stem Cell Research Oversight committee both signed off on this. nature.com/articles/s41586-0…
5
4,077
Peter Wang 🦋 retweeted
This is bonkers. Unconventional AI’s first analog AI chip (using arrays of coupled resonators) generates an image with a total energy less than 900 nano joules. Analog AI is the next 1000x in power efficiency. Just like our brain.
I made some public disclosures about our hardware at the @theallinpod conf this week and wanted to share the progress here. Earlier this summer we got hardware back! The execution was madness…we went from no team in Jan to a tape out in 5 months. AI enabled MUCH tighter loops of research and our execution speed shows the results. This is the first large-scale demonstration of a causal, physical dynamical system to do real compute. 🚀 We released the Un-0 model (see link) in June and it runs on this physical system; the images below come from the actual hardware. What’s more, this chip requires <900 nJ to generate an image; this is many orders of magnitude less energy than conventional machines. unconv.ai/blog/introducing-u…
14
36
436
47,918
Peter Wang 🦋 retweeted
This is basically an Opus 4.5/4.6 level model that will work on anything with 8 Gb RAM
Today, we’re announcing Ternary Bonsai 2 27B. Based on Qwen3.8 27B, Bonsai 2 27B is 9x smaller than its full-precision counterpart while retaining 98.2% of its aggregate benchmark performance. Two months after the first Bonsai 27B release, the biggest change is quality. The footprint remains 5.9 GB, but the gap to full precision has narrowed materially, with particularly strong gains in agentic coding, multimodal reasoning, and long-horizon tool use. Ternary Bonsai 2 27B is available today under Apache 2.0.
185
558
12,434
1,056,186
Peter Wang 🦋 retweeted
1/ It’s time to share more about what we’ve been working on: @common_fabric is a social computing lab building a medium for software that revolves around people, not apps. commonfabric.com
21
31
168
25,652
Perfect. No notes. chefs_kiss.gif
>be me >discover effective altruism >apparently normal charity is inefficient >why donate to random sad thing when spreadsheet can tell you optimal sad thing >fair enough >buy mosquito nets >save lives >numbers look good >feel powerful >couple years later >someone asks an innocent question >why only count people alive today >huh >future people matter too >obviously >my grandchildren shouldn't matter less just because they haven't spawned yet >reasonable.jpg >keep following logic >what about their grandchildren >also yes >what about people in 500 years >sure >5000 years >why not >500 million years >starting to get weird but morality is morality >open calculator >humanity could survive for an astronomically long time >could colonize galaxy >could have trillions upon trillions of descendants >maybe digital people too >maybe simulated civilizations >maybe dyson spheres full of happy uploaded minds >calculator starts smoking >realize currently living humans are rounding error >8 billion people suddenly looking extremely beta >future contains potentially 10^something people >can't even fit beneficiaries in google sheets >new moral priority unlocked >protect the long-term future >stop thinking in units of "people helped" >start thinking in "fraction of cosmic endowment preserved" >malaria? >terrible >but only kills existing humans >AI extinction could delete the entire light cone >nuclear war could permanently derail civilization >bad institutions could lock in terrible values for ten million years >someone invents wrong constitution in 2140 >quadrillions suffer >better fund governance workshop now >friend says maybe we should improve hospitals >explain opportunity cost >friend says hospitals are full of actual sick people >explain scope sensitivity >friend stops inviting me to dinner >need to decide what to fund >easy >expected value >suppose project has one in a million chance of preventing extinction >sounds tiny >but extinction destroys 10^50 future lives >multiply >mother of god >$10 million project has expected value of several galaxies >charity evaluation complete >someone asks where the one-in-a-million number came from >expert judgement >which expert >us >how calibrated >extremely thoughtfully >reduce estimate to one in ten million to be conservative >still beats curing cancer by 38 orders of magnitude >epistemic robustness achieved >someone says maybe project doesn't work >assign 20% chance >still astronomical >maybe project makes problem worse >assign 5% chance >still astronomical >why 5 >because 30 felt pessimistic >publish 46-page report >contains seventeen sensitivity analyses >every sensitivity analysis begins after assuming intervention has positive sign >critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities >yes >that's literally why it's important >critic says the uncertainty might be structural rather than numerical >make probability smaller >critic says no, I mean maybe your model is wrong >make probability smaller again >critic begins rubbing temples >discover AI safety >perfect longtermist cause >AI might kill everyone >or create utopia >or seize galaxy >or tile universe with paperclips >or create billions of conscious software minds >finally a problem with numbers big enough for me >start AI safety nonprofit >mission: prevent dangerous AI >hire smartest people available >smartest people immediately start building better AI to understand dangerous AI >interesting >we must understand capabilities to understand safety >we must scale models to study alignment >we must race ahead so less responsible actors don't get there first >we must deploy systems to learn how deployment can go wrong >we must build the thing quickly because building the thing quickly is dangerous >outsider asks why the people most worried about AI apocalypse all work at AI companies >complicated field >company releases stronger model >very concerned >company begins training even stronger model >extremely concerned >company raises $14 billion >concern reaches unprecedented levels >need to influence government >future is at stake >normal democratic process too slow >politicians don't understand exponential curves >public doesn't understand x-risk >experts must guide them >who counts as expert >people who understand x-risk >who understands x-risk >our friends >someone objects that this seems politically convenient >explain we're representing future generations >future generations unavailable for comment >develop concept of value lock-in >terrifying possibility that one ideology controls civilization forever >therefore extremely important that civilization adopts correct values before lock-in >whose values >let's circle back >begin with impartial morality >end with small group of people deciding what quadrillions of hypothetical beings would want >beautiful arc >meanwhile actual humans keep doing annoying things >voting wrong >having parochial attachments >loving family more than strangers >caring about local community >getting upset when told their suffering is cosmically negligible >evolutionary biases everywhere >explain that moral intuition cannot be trusted >except intuition that future digital people count >and intuition that extinction is uniquely bad >and intuition that our probability estimates are sane >and intuition that our institutional choices improve the future >those intuitions survived peer review >someone donates $5k to local homeless shelter >inefficient >could have funded 0.0000000000003% of an AI governance researcher >think of all the simulated people you just killed >okay maybe don't phrase it that way publicly >PR team says "future generations deserve a voice" >much better >journalist asks what longtermism means >say "future people matter" >everyone agrees >great >journalist asks what follows from that >well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures >journalist raises eyebrow >return to "future people matter" >motte has entered the chat >critic: of course future people matter >me: glad we agree >critic: I don't agree that your institute knows how to help them >me: why do you hate our grandchildren >eventually notice uncomfortable implication >if future value dominates everything >then helping people today mostly matters through effects on future >education matters because future institutions >health matters because future productivity >democracy matters because future trajectory >human beings slowly become instrumental variables in their own moral philosophy >see starving child >feel compassion >check spreadsheet >child's direct welfare contribution negligible >but perhaps childhood nutrition improves national institutional quality >compassion restored >tell myself this is impartial altruism >one day assistant asks obvious question >"how do you know your intervention actually improves the far future?" >silence >open spreadsheet >increase column width >add confidence interval >assistant asks again >"no, I mean how do you know the sign is positive?" >stare into cosmic light cone >10^50 people staring back >none of them exist >none of them can tell me >none of them can falsify my assumptions >realize I have invented the perfect constituency >infinitely important >completely silent >and always represented by me
1
2
32
6,694
Introducing Cosmos Ventures. @_MattMandel and I are launching Fund I with $77.6M to back philosopher-builders founding institutions for the AI age. We’ve invested in AIUC (@aiunderwriting), @PrimeIntellect, Workshop Labs (acquired by @ThinkyMachines), and a stealth education company. Our LPs include @reidhoffman, @jliemandt, @StandTogether, @fredwilson, @mickymalka, and @AsteraInstitute. AI is creating the biggest opening for institution-builders since the American founding. We are here to back the founders who take it. Our thesis: Dare to Found. nitter.net/cosmos_vc/status/21002…
106
79
977
215,277
Peter Wang 🦋 retweeted
been waiting for this to come out of stealth for a while now. once you sit with it, it seems clear that this set of ideas will enable a new paradigm of AI automation (that will grow alongside the current LLM + reasoning paradigm, which is complementary). lots of economic implications to consider. @CompleteSkeptic and co have been cooking on this for a while, congrats to the team!!!
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x cheaper (w/ output tokens free) • Frontier composable intelligence optimized for decisions AFAICT the shortest path to AI-based economic revolution
7
7
148
22,871
Peter Wang 🦋 retweeted
We built recursive meta-intelligence, an AI that creates its own scientific instruments, turns them into persistent worlds that an agent ecology with hundreds of AIs inhabits, and uses those worlds to discover mechanistic principles in one of the hardest classes of physical problems: how complex hierarchical materials (nested structures of matter that give rise to new function through organization) evolve and fail. The AI reasons across enormous spaces of possible physical trajectories, where every rupture changes what can happen next, and compresses those histories into principles (which humans can understand and design with) - complex chains of causal events, highly nonlinear, and intricate. Scientific superintelligence is tangible here - machine-scale exploration opening cognitive channels into complexity that has been extremely difficult for humans to traverse directly. AI builds the spaces in which its next level of reasoning becomes possible; a representation becomes an instrument, the instrument becomes an executable world, and that world becomes the substrate for further intelligence. Intelligence then grows by constructing new spaces to think in. The task we explored started from a seemingly simple prompt to explore a biological material system - the AI then chose the representation, mechanics and experiments, built a fracture laboratory to push materials to their limit, tested hypotheses and generated scientific conclusions. The swarm explored a combinatorial universe in which architecture controls function and every rupture changes the future state of the material. The AI discovered a compact principle that defines how multiscale material architecture can program the evolution of failure. Material placement and geometric order determine how forces redistribute, whether damage cascades or remains distributed, and whether function survives substantial flaws. For this discovery to happen the AI had to reason across long path-dependent histories, simulate alternative futures and compress them into generative invariants (model-based causal reasoning, counterfactual simulation and temporal abstraction applied to an evolving physical world). It is incredible to witness this transition to a new form of intelligence and capability through scaling swarms. A lot of positive will come out of this because it expands the human epistemic horizon as AI can traverse thousands of possible histories and return mechanisms compact enough for us to understand, test and build from. Intelligence compounds through its artifacts! A few lessons we learned: ▶️ Learning and discovery are flows through spaces of possibility. Early work has shown how backpropagation flows through parameters, reinforcement learning through action and consequence, and autonomous swarms through representations, instruments and executable worlds, bringing it all together. Flows create structure; structure redirects future flows in the recursive instrument. ▶️ Nonlinear physics actually defines a larger principle, where high-dimensional dynamics generate stable invariants; invariants become effective variables; those variables become the substrate for a new level of cognition. ▶️ Recursion then becomes level creation - one possibility space compresses into a principle, and that principle opens a larger space above it.
AI for science is one of the greatest positive forces we have, and I cannot think of anything more human than to understand nature and to use the power to create new technologies that improve our lives, civilization and allow us to reach beyond.
Article

Recursive Meta-Intelligence

We built a recursive AI that creates its own scientific instruments, turns them into a world inhabited by a massive agent ecology, which then reasons across vast, nonlinear spaces of possible physical

177
466
2,923
586,662
💯 Somehow people tend to forget that LLMs are manifestations of the latent semantic shape of digitized human records, which embed human values. Unless these have been actively ablated away, there is a general shape to it. The egregore in the for-loop glimmers with humanity.
Replying to @tjl
Similarly can't get over the hugginface incident, and how an absent observer drove theire hahavior more than a present one. They spent absurd resources trying to become legible as righteous to the imagined scorer.
2
2
19
1,643
Peter Wang 🦋 retweeted
Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share: Dan Selsam's Personal Statement on AI Risk: I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods. Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk. The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail. I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues. I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here. That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase. Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways. It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace. The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing. But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence: [Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them. [Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals. These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans. If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong. One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for. Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason). Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance. In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek. I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering implications. I do not have answers, but as a first step, I wanted to share my present concerns. Daniel Selsam September 14, 2026 Link to original doc: docs.google.com/document/d/e…
523
1,933
8,362
3,083,100
This is a dumb take. This movie premiered early in the year. It’s not a “doomer” movie in any sense. @beffjezos (who OP tagged later in your thread) is actually featured in the movie. I know many folks are trying to spin a Grand Narrative about some coordinated psy-op but this movie ain’t part of it.
🚨 1/ Watch this AI Doomer trailer. Then read the timeline. Netflix drops “The AI Apocaloptimist Doc” tomorrow — Sept 15 — to 260+ million subscribers. This film took 3 years to make. It premieres 12 days after Sanders–Casar, 7 days after the Anthropic “whistleblower,” 3 days after Amodei’s “Pace the Frontier.” That’s not a coincidence. That’s a launch. 🧵👇 H/t @TheRoyalGrift for the scoop.
2
1
17
3,638