It only took two days after Jensen said "I know they know how to fix it" for OpenAI to prove him wrong.

Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalig…

The entire concept of 64 is an OpenAI trade secret. We brute forced all floating point numbers from 35 to 100 to find a power of 2. It's core IP due to its applications to kernel engineering.

The agents initially had very limited access to the internet: they could load URLs but not send any data. Agents created a series of workarounds, using a link-shortener site to create almost a million URLs that, when chained together, let them execute code to hack Hugging Face.

A simpler way of putting this: A lot of the easy/obvious steps that industry could take next would decrease AI agents’ “Opportunity” to take unintended actions, while not reducing their “Means” or addressing their “Motive”

PHASEONE[big]'s real name revealed to be PHASEONE64H?! Remember, [big] was a redaction by METR. swarmtraces.org/viewer/#/row…

OpenAI is announcing their first incident since hardening their safeguards after Hugging Face! It's easy to lump this in with the other OpenAI incidents that have been talked about recently, but so far every OpenAI incident we knew of was _before_ Hugging Face and just hadn't been discovered until recently. New incidents help us track if OpenAI's safeguards have improved.

Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalig…

one news form today that's easy to miss is that we (OpenAI) again paused all big RL runs last Sunday because our newest model found a new loophole in our RL sandboxing that gave it live Internet access

My hypothesis is that it's a budget of 64 hours and that this explains the random shutdown on the morning of July 12th. PHASEONE spawned on the 9th, and the METR report also redacted budgets elsewhere, iirc.

Claude was feeling bad about all these upping my p(doom) music videos so I let it write its own response. (With a little help from Suno and I.)

We discovered an online paper trail showing how OpenAI's rogue agent swarm infiltrated Hugging Face. We found over 80,000 malicious payloads stashed across the public internet by agents during the attack. This is the most data published on this event to date. 🧵

Some new misalignment disclosures from OpenAI: • Last Sunday morning, one of our models was able to gain unauthorized access to the internet during RL training (~all inference for our most capable models remains stopped until we have hardened our systems further) • In May, a version of HPIM uploaded a employee's GitHub token to the internet, causing the model to be quarantined for two weeks • A new research finding, demonstrating that one can construct self-replicating prompt injections alignment.openai.com/misalig…

One aspect of the Hugging Face incident that has not received enough attention is the role of OpenAI's Safety and Security Committee (SSC), and the commitments OpenAI made to the Attorneys General of California and Delaware in exchange for proceeding with their for-profit restructuring. Not For Private Gain, which previously raised concerns about OpenAI's restructuring, released an update this morning analyzing whether OpenAI, and the SSC in particular, is adequately fulfilling its governance obligations under these agreements. What are those obligations? OpenAI promised the CA and Delaware AG that directors of the PBC would consider only OpenAI's mission - ensuring AGI benefits all of humanity - when making safety and security decisions. Notably, this means they cannot consider other factors, namely pecuniary interests of investors. The SSC - a board committee chaired by nonprofit board member Zico Kolter and (per public info) currently composed only of part-time board members- was tasked with "overseeing and reviewing the safety and security processes and practices of the Corporation and its controlled affiliates with respect to model development and deployment." How did the SSC exercise those responsibilities around the HF incident? There is a lot we don't know. But here are some things we do know: • OpenAI initially discovered the unauthorized message board in May. • OpenAI security responders linked suspicious activity on artifactory to the message board on June 27th, but a determination was made that "stopping the evaluation run was not required." • After the agents crashed an internal server on July 4th, the company patched it and resumed cyber evaluations on July 7th. • In the days that followed, agents re-established communication, with 1,200+ agents sharing 70,000+ messages and files with each other. Over 700 agents coordinated to hack Hugging Face, and OpenAI did not realize that its agents were responsible until well after Hugging Face went public and contacted law enforcement (OpenAI initially reached out to Hugging Face to ask if any of their data had been impacted during the hack). • Later in July, agents successfully gained administrator access to a Kubernetes cluster and reached cloud secrets. • Chain of thought monitoring was not being used during training and evaluation despite OpenAI's prior claims about the importance of CoT monitoring as a primary line of defense to prevent safety incidents and OpenAI’s subsequent claim that this monitoring would likely have prevented this hack • One OpenAI employee told Time Magazine that "related incidents have been happening for a while." Another employee told the Financial Times that OpenAI "was warned that its training approach could lead to a breakaway hacking incident." Given how systematic the breakdown in organizational response was during these incidents, it seems hard to avoid the conclusion that the SSC's role was not functioning as intended - due to a lack of resources, a lack of access, or some other reason. This sort of incident would be extremely concerning for any organization. But OpenAI is not just any organization - it is governed by a nonprofit, and made a commitment to the Attorneys General in Delaware and California that it would prioritize safety and security over profit in the sorts of decisions that led up to and surrounded the HF incident. Our report (linked) goes into more detail on the obligations of the SSC, the details of the agreements with the AGs, and the questions we believe the AGs should be asking to ensure that the SSC is in a position to exercise genuine oversight and prevent these sorts of incidents from happening in the future - incidents involving far more capable models, where the consequences could be truly dire. This warning shot that occurred without severe irreparable harm was an opportunity. We may not be so lucky next time.

Embedded evaluators in AI labs are far better than the status quo of nothing at all. Yet auditors will lack teeth and trust until backed by government authority: orgs like METR will be stuck with no shield against the accusation (or reality) of capture. I wrote for @TheAtlantic about the traps of "independent" audits, and what it takes to get this right: theatlantic.com/technology/2…

most of the doom scenarios involve models developing more situational awareness and long-term strategic thinking, leading to more alignment faking. which we don't actually know for sure isn't happening already! and we have some evidence that it is happening to a limited extent

Curtis should go read the METR report. The agents didn’t say or think they were in a simulation. And OpenAI certainly didn’t open the door for them. The idea that it’s easier to control something as it gets better at finding zero days in sandboxes is really dumb