Policy @bfl_ai. Affiliate @BKCHarvard. ex-Stability AI (weights), GoogleX (drones), Uber (rides), Coinbase (magic beans). Views my own
United States
- Tweets 771
- Following 322
- Followers 2,223
- Likes 1,790
Pinned Tweet
Nearly every AI firm is invoking frontier risk to justify a persistent gap between open and closed models. But this application of the precautionary principle deserves more scrutiny than it gets.
Restricting access to useful technology—models that will, in their developers' own words, transform the economy—shouldn't be our primary response to uncertain risks.
Check it out at @aif_media! There are already a bunch of reasons firms might not release their best models openly: cost recovery, competitive pressure, anxious investors. But we should be skeptical of efforts to freeze open-source behind the frontier under the guise of risk.
2
2
15
4,643
My audition for Karoline Leavitt: "Open weights are a right, not a privilege. Next question."
A crazy week of meetings in DC. To sum up the vibes across Congress in a single quote:
"There is lots of affection for open source here. But the ecosystem is going to be absolutely steamrolled if they keep saying 'don't do anything' ".
And a timely opportunity to speak at Freedom Tech DC alongside @matthew_d_white and @matthew_pines about the importance of open source and open science! We need to continue thoughtfully bridging the Bay-Beltway divide.
3
3
28
2,571
"We have to make sure that the models you can run in a disconnected environment are substantially less capable".
I've spoken for years about why open weights shouldn't be a poor cousin of closed source by force of law. It's disappointing to see this reasoning making a comeback.
We're used to thinking of open-source models as an unadulterated good. But in the case of AI, they can actually pose additional dangers, as @ReidHoffman and I got into at #CGI2026. I appreciated this nuanced discussion.
Community note
All recent large-scale cyberattacks have been performed by proprietary AI models from OpenAI and Anthropic.
No evidence that open-weight models present any additional cybersecurity risks.
nytimes.com/2026/09/23/tec…
anthropic.com/news/investiga…
en.wikipedia.org/wiki/OpenAI%E2…
opensource.org/blog/openness-…
2
15
858
Ben Brooks retweeted
Fantastic essay about Chinese AI labs:
“If there is one thing I’d want to delete from the American discourse, it’s the phrase “China’s AI strategy.”
Every single meeting, be it labs, investors, media, all independently, without my prompting, described domestic competition as more brutal than competing with American labs. When I described the US framing of Chinese AI as a coordinated bloc moving at the direction of the state, people laughed. Not defensively. They found it funny. I was called a stupid ABC (American-born Chinese) more than once because of this.
These labs are actually far less coordinated than US labs are, which should have been obvious to anyone who watched the previous cycles of Chinese tech. I said no analogies, but I’ll allow one because it’s theirs. The last cycle had a name: 百团大战, the War of a Hundred Groupons (it was actually 5000). Hundreds of group-buying companies raised money, undercut each other into oblivion, and Meituan emerged as the victor. This cycle already has one too: 百模大战, the War of a Hundred Models.
What actually exists is an ecosystem of labs relentlessly undercutting each other on price, stealing each other’s researchers, raising against each other, and trying to win a local arms race. Hundreds of labs and neolabs have raised, died, and been replaced, continuously.
The things we experience in the US as hostile acts are mostly the exhaust of that domestic fight. The distillation attacks and the collapsing token prices are not a strategy aimed at American incumbents. They are what a hundred companies do to each other when none of them can win and none of them can stop.”
Everyone has a take on China and AI right now. Very few have actually been to China.
I spent last week in Beijing and Shanghai meeting most of the major model labs, researchers, VCs, and founders building Chinese AI. This trip meant a lot to me beyond the work. My parents were born and raised in Beijing. I spent my earliest years growing up in my grandparents' Xicheng district apartment. As a child, I wanted to be a diplomat because I thought US–China would be the defining relationship of the next hundred years. I got as far as spending a decade investing across both and then the two halves of my identity stopped being compatible. This was my first trip back since.
Below is a write-up of what I found. In short: there is no version of the next decade where Chinese open models don't matter.
earnedintuition.substack.com…
ALT At Tsinghua University, Beijing
31
176
1,221
158,409
SOTA performance, < half the parameters, 4x the speed--and open weights.
Introducing FLUX 3 Action.
An open weights 7B World Action Model that achieves first place on the RoboLab benchmark.
It outperforms the previous best open model by 6.1 percentage points while using 56% fewer parameters and running up to 3.95x faster.
FLUX 3 Action removes the usual trade-off between world action model performance and VLA speed: it still predicts video and actions together, but plans more than twice as far ahead and runs faster per second of robot motion than the strongest open VLA.
Teams can fine-tune FLUX 3 Action on their own demonstrations to create policies for a particular robot and task. Together with @nvidia, we also integrated FLUX 3 Action natively into @huggingface's LeRobot, with fine-tuning recipes included and edge deployment on NVIDIA Jetson.
Beyond robotics, we’re also seeing promising results training task-specific policies for acting in simulated environments like gaming, controlling a vehicle, computer use, and wherever else a model needs to understand a visual environment and then choose what to do next.
FLUX 3 Action builds on the same image, video, and audio pretraining as FLUX 3, but uses a smaller architecture designed for practical deployment. In midtraining, we trained the model to predict actions and future frames together.
We’re releasing the weights, code, fine-tuning recipe, benchmarks, and reproducible examples so researchers and developers can build on the model with their own robots, environments, and tasks (see below).
5
408
History is repeating itself, so let's repeat the history.
SB1047 had a kill switch provision. After much furor, Sen. @Scott_Wiener went to great pains to stress that it applied only to models in the custody of a covered entity. He made amendments to that effect.
I was reassured that, at least in relation to kill switches, his team understood why a "full shutdown" functionality isn't compatible with open models. They didn't want to cripple open development with manifestly unworkable obligations.
But Gov. @GavinNewsom vetoed SB1047 on the basis that, inter alia, "smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047". This is a view widely shared in the halls of Sacramento and Washington.
If we revisit a Californian kill switch, there will be immense pressure to apply those requirements to models both at and behind the frontier. There'll be similar pressure on the AI Kill Switch Act from @tedlieu and @RepNateMoran, which requires covered entities to ensure they can "stop inference", "terminate user access" and "shut down" models.
The net effect is you can develop a capable model if you release via API, but not if you release via open weights. As I and many others have laid out before, that is a terrible outcome that will feudalize the entire US economy around ~three Bay Area firms.
If anyone is permitted to develop capable models (whether behind, at, or beyond the known frontier), our regulatory settings should promote responsible diffusion, not enforce a policy of containment behind paywalls.
BREAKING: I just signed an EXECUTIVE ORDER to dramatically accelerate independent oversight of AI companies and advance the creation of an “AI kill switch.”
With Donald Trump and Congress failing to lead, California is stepping up to keep all Americans safe.
1
1
12
860
Ben Brooks retweeted
Over 2 years ago @sayashk and I wrote a detailed deconstruction of p(doom) and argued that its primary function is to launder vague, evidence-free intuitions and fears through a facade of quantification. It remains 100% relevant today. normaltech.ai/p/ai-existenti…
There is a strange laundering of this number. It was floated by Anthropic employees without public justification. Then mathematicians pick it up and now it is attributed to their judgment.
It’s fine to survey AI practitioners, but posts like this are a bad epistemic practice.
12
75
355
52,491
Confidence and conviction aren't the same thing. This exchange today between Coxon and the NYT reveals a lot about the bizarre conflation of the two this summer / decade:
> NYT: [The claim is] no other activity poses this level of danger. A lot of people want to understand how exactly AI can pose this kind of risk to human existence. Can you play that out for me, as concretely as possible?
> Coxon: The concrete scenario is actually a tough question... People have apps on their phone that control physical devices, like household appliances. ChatGPT and Claude can access these appliances over the Internet. You can imagine AI is tricking people into doing things. So it would be pretty easy for a future version of Claude to hack into a military drone and have it fly around killing people... I struggle to visualize the ending viscerally enough to keep me up at night.
This is conviction, not confidence.
To be clear, this scenario isn't the steelman for catastrophic risk; @Nataliekitro probed him intensely (at times incredulously); and Coxon, a talented researcher, most likely has a richer mental model for catastrophic risk than what was conveyed in this short podcast.
Yet extraordinary claims with prayers for extraordinary intervention require extraordinary evidence. This isn't enough. The public should demand more than hand-wavy projections inviting us to leap from the Hugging Face attack to the end of the world. For all the ink spilled this week, we actually ask far too little of those advocating policies to pace / condition / outlaw / nationalize frontier development, which are at best anti-competitive and, at worst, deeply anti-promethean.
1
6
819
I get nervous when NYT, Fox, and BBC start live tweeting AI policy updates like it's the Strait of Hormuz. The chance of a regulatory misfire is high, especially when frontier labs are out endorsing mandatory kill switches today.
1
5
162
FAS superforecasters predict a 1% chance that a nuclear weapon catastrophe kills 10M people in 20 years. 1%/10M/20y. That's the nuclear equilibrium. On that basis, we enforce a policy of denial, not a policy of responsible scaling, pacing, or paywalling. The US:
> Bans development of nuclear weapons without presidential authorization on penalty of 25 years to life.
> Bans sales or transfers other than to designated agencies on penalty of 25 years to life.
> Compensates authorized contractors on a fee, not equity or profit sharing, basis.
> Wages war to prevent development, acquisition, or use of WMD.
If current frontier lab workers earnestly believe they have a 10% chance of killing everyone in 10 years (10%/8B/10y vs. 1%/10M/20y), none of the policy proposals recycled through the blogopshere and NYT this month truly rise to the occasion.
1
4
402
Pacing the frontier isn't inherently anti-open. It may be anti-fast, but it isn't anti-open.
But the $2T unresolved question that goes to the heart of open weights is: what are the "Y" conditions at the frontier, who decides, and does "X" capability actually correlate with risk?
Neither Coxon, OAI-HF, Fable, nor chikunyuga virus have changed the contours of this debate from 5-7 years ago.
1
2
7
501
I deeply respect the Pope, but we should be mindful of the Church's legislative instincts for technology diffusion.
"The art of printing can be of great service insofar as it furthers the circulation of useful and tested books, but it can bring about serious evils if it is permitted to widen the influence of pernicious works.
It will, therefore, be necessary to maintain full control over the printers so that they may be prevented from bringing into print writings which are antagonistic to the Catholic faith, or which are likely to cause trouble to believers."
Alexander VI to the dioceses of Germany, 1501
Sound legislation should encourage scientific and technological creativity while safeguarding fundamental human rights and freedoms. It should protect users from exploitation, preserve personal privacy, ensure transparency in the use of emerging technologies and guarantee that they strengthen democratic institutions. vatican.va/content/leo-xiv/e…
1
7
658
An unnamed official is mooting pre-release controls on open models past a certain threshold.
Just my friendly tri-annual reminder that the Biden Diffusion Rule exempted models from export control if they were released openly.
BIS dropped that Rule in 2025 because it "stifled... saddled... undermined" American innovation.
Hard to square the circle.
2
4
487
The NYT gets open weights! I love to see the press amplifying the voice of actual developers:
Using Chinese models was like owning a house while using U.S. models was like renting one, they said...
“I don’t think people should worry so much about who built it,” he said. “The focus should be, does it deliver the capability you need?”...
Developers are building systems controlled elsewhere, which means the technology can be yanked, altered or re-priced without warning.
5
6
23
2,237
From latent diffusion to FLUX 3, it's been a wild ride 🥹 Open weights coming soon!
FLUX 3 Video is here.
Serious, fun, creative, real, cinematic, whatever you need it to be.
Native audio, Text to Video, Image to Video with multiple frames, video continuation, dialogue in multiple languages.
Comes with Draft mode so you can explore ideas fast at a fraction of the cost.
Up to 20 seconds and 1080p native.
Available in the API or in your favorite tool. 2K, 4K, and Open Weights coming soon.
8
406
Ben Brooks retweeted
I have been really looking forward to this one personally, BFL has a great team. Some of my mutuals in early access like @dreamingtulpa have done amazing things with it. 20 second clips in 1080p, and they say it beats all SOTA in text-to-video on their internal evals.
FLUX 3 Video is here.
Serious, fun, creative, real, cinematic, whatever you need it to be.
Native audio, Text to Video, Image to Video with multiple frames, video continuation, dialogue in multiple languages.
Comes with Draft mode so you can explore ideas fast at a fraction of the cost.
Up to 20 seconds and 1080p native.
Available in the API or in your favorite tool. 2K, 4K, and Open Weights coming soon.
15
9
186
19,264
One characteristic of anti-open hysteria (open source, open weights, open data, open web) is a tendency to ignore the massive diffuse benefits, and concentrate on limited acute harms. Deeply condescending to describe the beneficiaries of openness as "bored 19 year olds".
1
6
446
In their defense, when this debate really kicked off in 2023, Anthropic was basically the only frontier lab that wasn't pushing some cockamamie scheme to "license model developers" or "send in the AI peacekeepers" or whatever.
Don't get me wrong, their "FDA / FAA for weights" is still a bad idea. But Anthropic has been commendably consistent about what it believes, and how it would weigh the competing interests.
There’s been a lot of speculation about where we stand on open-weights models. We’ve outlined our views in full here: anthropic.com/news/position-…
10
1
93
35,695
The new House FRONTIER Act has the same old problems as before.
Public authorities have a hard time deciding when the benefits of drugs, vehicles, and financial products outweigh the risk (not to mention the Internet). Each of these has decades / centuries of actuarial history. But under this bill, the federal government would also need to approve the methodologies that auditors will use to determine the acceptable catastrophic risk posed by 1e26 FLOP models, which have ~1 year of actuarial history.
Two likely outcomes:
1. These methodologies are conservative. They overweight speculative risks and underweight diffuse benefits. Auditors cry wolf, obtaining emergency restriction orders at the slightest hint of a capability breakthrough or offense-defense lag. It becomes difficult to release a frontier model publicly via API, and impossible to release one open-weights.
or
2. The Department of Commerce is mindful of that ^ possibility, and gives auditors wide latitude to determine their own methodologies. Standards are lax, developers can forum shop, and nothing really changes from the status quo.
If you believe that AI is an important technology with dual-use properties, either scenario would seem to be undesirable.
This is the problem with proposals that go beyond transparency and try to prescribe acceptable risk thresholds. They jump the gun. We simply do not know enough about the risk-utility profile of this technology to form a consensus view of acceptable catastrophic risk. Drawing a new line in the sand could chill the widespread release of legitimate technology (whether through export controls, liability reforms, or these private-audits-with-public-licensing proposals).
To be clear, in my view, a regulatory determination of acceptable catastrophic risk is vastly preferable to the tortious approach endorsed by e.g. the original SB1047 / RAISE Act. Regulatory thresholds can be inspected, contested, and adjusted based on broad public input. State jury verdicts cannot so easily.
But even so, we are way over the skis here.
You wouldn't guess it from all the "light touch" rhetoric, but the Obernolte-Trahan AI bill is uncomfortably close to FDA-for-models.
CAISI will license auditors, who must verify the "adequacy" of the developer's safety framework for achieving an "acceptable" levels of risk. But if CAISI doesn't agree with the standard or methodology, it can revoke the auditor's license.
These auditors aren't verifying compliance with the developer's safety framework. They're verifying compliance with a hand-wavy, open-textured standard at the discretion of a federal agency.
That goes beyond what even the most interventionist states have enacted.
1
1
5
846