Third-party evaluators mostly test frontier AI models through an API. That misses the risks that come from how companies build and use AI internally. Our new paper makes the case for embedded assessments. It was led by Jacob Charnock. The other co-authors are Sophie Williams, @zaheedkara, @Manderljung, Alejandro Tlaie Boria, @StephenLCasper, @AnkaReuel, and me. 🔗 Read the paper: governance.ai/research-paper…

An AI agent hacking into another company’s systems seems like something policymakers would want to hear about. Yet there are currently gaps in the legal requirements to report incidents like this. After OpenAI agents breached Hugging Face, California officials said the incident “did not meet the threshold” for mandatory reporting. OpenAI notified EU authorities about the attack, but the EU Code of Practice does not require independent external investigations or routine sharing of the lessons learned across industry (nor do any US state laws!). In a new @GovAIOrg policy brief, I look at how we could close these gaps in US state and EU incident reporting regimes. Broadly, these regimes could require broader disclosure to bring more incidents to light; establish rules for triggering and facilitating independent investigations to understand what went wrong; and require corrective action and information sharing across industry to prevent similar failures. I recommend seven changes to this effect: 1. Expand mandatory reporting to include near misses. Incidents can reveal important safety failures, even when no serious harm occurs! These near miss events give developers a chance to address those failures before a similar incident actually causes damage. 2. Cover incidents during training, evaluation, and internal use. As seen in recent incidents, AI agents can affect external systems even before they are publicly released. Reporting requirements should cover serious incidents and near misses involving models still under development or never intended for release. 3. Require independent investigations of sufficiently serious cases. Internal investigations are useful, but they may overlook how developers’ own decisions, safety practices, or organizational culture contributed to incidents. For serious incidents (or near misses thereof), independent external investigations can provide scrutiny of developers’ explanations and identify additional failures. 4. Make actions traceable to the agents responsible. Attributing actions to a specific agent is an important step in learning about an incident because identifying the context in which an agent is operating (its instructions, permissions, and operating conditions) can help explain what went wrong and why. Developers should be able to reliably trace actions within their own infrastructure and on external systems to the specific agent instances responsible. 5. Specify what evidence developers must preserve. Adequately investigating incidents requires access to evidence. But evidence can be lost before an investigation even begins (e.g. logs can be overwritten, models updated, and operating environments deleted). Developers should retain essential records for defined periods and protect relevant evidence as soon as an incident is suspected. 6. Pilot conditional safe harbors to encourage candid, early reporting. Developers may hesitate to report incidents if doing so exposes them to penalties for safety violations. Conditional safe harbors could offer limited relief for inadvertent violations when developers report promptly, preserve evidence, cooperate with investigators, and correct the failure. 7. Require corrective plans, check whether fixes work, and share lessons across industry. Findings from one incident could help prevent similar failures at other companies. Developers should explain what they will change and by when, regulators should check whether those changes work, and useful lessons should reach others relying on similar safeguards. Incident reporting regimes can be an incredibly useful tool to prevent future harm. To do so, reporting an incident should lead to a process seeking to establish what went wrong, fix the failure, and share those lessons with other relevant actors. This way, each incident can leave developers, policymakers, and society better equipped to prevent the next one.