This. I would just note that liability alone can’t handle the tail risks Bessent mentions, as labs might be bankrupt before they can be held responsible for the full harm they caused. Insurance mandate helps address this. It also gets us embedded inspectors who are hired by insurers, rather than just METR or other anthropic friends

An antitrust shield without government help

If only there was a way to get auditors with good incentives… ai-frontiers.org/articles/do…

And they said an actuarial model wasn’t something to get excited about There will be watch parties all around the world on release day

🧵

Ah yes, an “FDA for AI”. Because we can just run randomized controlled trials on agent swarms What could go wrong when evaluating Highly Persistent Internal Model for Safety and Efficacy

It seems to me like the ideal model for AI governance would be embedded auditors from the government, perhaps from a combination of CAISI, the national labs, nat sec world, etc. who don't have any direct regulatory or other coercive authority, but who can flag things to the executive for emergency concerns and who generally have good awareness of what is going on. And then perhaps we have some kind of well defined procedure with careful judicial review to get temporary restraining orders or preliminary injunctions for imminent harms, medium term injunctions for fairly pressing harms but where we don't want to burden the companies for a super extended period of time (The FRONTIER Act does a version of this). And then all of this exists alongside the insurance infrastructure, and the insurers can use the data that comes from the national labs or CAISI or the Natsec world as they see fit in the same way that the Nuclear Regulatory Commission does a bunch of in-depth inspections, and a lot of that information is used by nuclear insurers to price risk. Another helpful inspiration from Nuclear world could be INPO, an industry consortium that also does detailed technical inspections that inform insurers. There are spillover benefits to safety research, which means frontier labs might be underinvesting in it because they can’t internalize the positive externalities. Allowing them to form an industry consortium to pool resources toward alignment and security research can help address this. But we shouldn’t make it a self regulatory organization, or give it any kind of binding authority. Alongside all of this, I’m open to IVOs that audit for 1) risks that are uninsurable because they’re too large of tail risks and 2) risks that aren’t really torts or are poorly handled by ex post liability (surveillance/leverage/persuasion by the labs, as an example). But this may not be worth it, as I don’t want to later on an insane amount of regulation.

I hear lots of AI policy folks say the problem with an insurance mandate is that we don't have time. “Timelines are too short” and while insurance might be theoretically good, in practice the IVOs might just be good enough. I’ve come around to thinking they might be correct. But I guess what I’d say is: let's not let this obscure the fact that the IVO design is deeply imperfect (race to the bottom), and the insurance design is much better, though still not a cure all. So the upshot is that we should implement whatever policy we can get in the immediate term while continuing to work towards good institutional design. That is, we shouldn't let the stopgap of FINRA or IVOs or whatever become sticky. And the fact that it probably will is a huge risk to be aware of

Very much appreciate someone laying out the stopgap plan that we urgently need. Our governance of AI can have stages: I remain skeptical of IVOs’ incentives, and eventually we’ll need something better-designed like a catastrophe insurance mandate for frontier labs. But IVOs could be helpful in the short term before they get captured or fall to bad incentives. It seems like a reasonable order of policies is: ASAP: -enabling the frontier labs to coordinate a slowdown by having the Admin promise narrow protection from antitrust scrutiny -the phone-call-based governance Anton lays out where the Admin tells labs they need to have METR, Redwood, etc. red team their systems Short term: -formalize that requirement for third party audits, probably by EO, whatever -Finra for AI or whatever. Not a particularly great design I don’t think, but better than nothing. -continued antitrust shield to coordinate a slowdown if needed “Medium” term (6mos - 1 year): -ideally and (very doably), get insurance ecosystem up and running to create the right incentives for third party auditors -if that can’t happen, IVOs are second best Medium term +: -we really should design institutions with better incentives than IVOs. Stopgaps are fine, but this technology could be too consequential to leave it up to “I trust METR to do a good job.” METR is great! But organizations change when you give them new jobs and incentives! Gabe Weil’s catastrophe insurance and liability work is the best idea i’ve seen for incentive design, and we will really need that. If for whatever reason (perhaps political) we can’t do that in the immediate future, that’s fine, I’ll take “Finra for AI” or IVOs or whatever. But we shouldn’t let those models become sticky! Eventually, we need those IVOs or whatever they’re called to be hired by insurance companies that actually have a financial motive to get the audits right. So to anyone who says “timelines are too short for insurance”: you’re probably right. That means doing non-insurance policies asap. But it doesn’t change the fact that ultimately insurance is better-designed than IVOs and we should implement a well-designed system with good incentives as soon as we can. We can do X, Y and Z (non of them insurance) to prevent “doom” in the super short term. But if it turns out we don’t all die in the next year, then we can do the right thing and do insurance! “Long” term (more than a few years away): -the world might be super changed, insurance won’t be enough, and we’ll need lots of ideas for different institutional designs -or maybe progress and disruption from AI will be less than expected. Even here, insurance isnt a cure-all, and we’ll need to do more than that. This scenario just means we have more time to do good stuff, and hopefully can do better than haphazard stopgaps

Oops. Should've said ~4 months

The reason that METR and Redwood are preferred for the Hugging Face investigation is that this incident is much more relevant for its alignment implications than its cybersecurity implications. Everyone gets hacked by external actors; we don't need crash public incident reports for most of those. Not everyone gets hacked by internally deployed models that also autonomously hack external partners. Accurate decomposition of agent behaviors and analysis of how OAI's deployment configurations and testing practices may have contributed to reward hacking/emergent misalignment are the key objectives of this investigation -- not description of where OAI's DLP, EDR, and reverse proxy systems were poorly engineered. But yes, METR, Redwood, and Apollo should partner more closely with/hire from Crowdstrike, Mandiant, Deloitte, etc., to upskill in cybersecurity (and vice versa).

Agreed that many of the mitigations for insurable risks happen to also mitigate uninsurable risks (like alignment research, good cybersecurity, and overall caution) I’d just note that insurance can in fact cover many “catastrophic” risks. We could mandate that frontier labs carry insurance for very large amounts, with the government providing a backstop as it does in many other insurance industries. However, instead of insurance covering only the first layer of harm and the government backstopping the tail, insurers and government could insure in parallel: for each dollar of harm up to, say, $1 trillion, insurers pay out 10 cents and the government 90. This caps insurer exposure at $100 billion, but because insurers pay 10 cents on every dollar, the premium they charge labs is 10% of what covering the full $1 trillion would cost. The government charges labs the other 90%—it merely multiplies the insurers’ premiums by nine.

Agreed with @The_DanielKing that the Under Secretary should be able to lower the compute threshold that defines a "frontier model." Additionally, to avoid a race to the bottom, two options: 1) auditors should be randomly assigned rather than chosen by the labs, or 2) we should require the labs to carry catastrophe insurance. The insurers would then have an incentive to conduct quality audits or hire auditors who can.

Call for books and movies: crime families but they're AIs "agents, with their novel set of characteristics (extreme cyber competency, ability to cheaply read a million words in seconds, persistence), will also probably change the contours of digital crime" hyperdimensional.co/p/on-the…

🚨 FAI is hiring for 7 roles across research, external affairs, and ops. $20/hr (intern) to $150 - 300k (sr. counsel). We've 3x'd this year and moved to Capitol Hill to be close to the action. Come build the future of think-tanking with us. 🧵 w/roles:

Some examples of tasks I would do as a paralegal that seem easy for AI now, but for which the AI tools were still very lacking 1 year ago: -making 3 or 4 page research memos with basic biographical information and links about judges and opposing parties -making copies of an image rotated many different ways, each on a different page, for showing a witness during a deposition -proofreading documents for grammar errors -no joke: taking concentration values (ppm) from a pdf from a laboratory and plugging them into a spreadsheet to calculate another value