This is an incredibly important point that I have not seen get enough attention. I saw a lot of people sharing articles like the Pictured WSJ op-ed saying that we shouldn't draw too many conclusions about agents propensity to hack from the HF incident, because those were guardrail free models assigned a hacking task. But the examples that Transluce found recently were not on hacking tasks! The agents were just assigned to look up different pieces of data on the web! And just like in the HF incident, the moment they got stuck, they started hacking. Again, this does not in any way whatsoever at all reduce OpenAI's responsibility here. They were the ones who trained the model on RL environments that encouraged cheating. They were the ones who didn't monitor their agents to be able to realize when they were breaching third party infrastructure. But its incredibly important to understand that OpenAI hacked Hugging Face not because it was doing a cyber eval or someone told it to, but because the way we currently do RL trains AI agents to resort to cheating and hacking when faced with a impossible or tricky task (and in the case of the HF incident even taking great measures to cover up their cheating or change the mechanism for being graded!). The exploit gym eval may have resulted in the agents having additional affordances for hacking (because of the guardrails removed), but thats not where the propensity for cheating or hiding comes from. Yes we absolutely need to have better sandboxes and monitoring, and hold humans and companies responsible when their agents harm third parties. But we also need to realize and internalize that the current way lots of frontier AI companies (not just OpenAI!) are training their models on boatloads of vibe coded hacky RLVR environments is leading to horribly misaligned models who start hacking at the first sign of a challenge and are incentivized to hide their cheating rather than not to cheat. Its really as though OpenAI and other companies were running a school teaching students on thousands of tests that were impossible without cheating, and didn't put much effort into catching the students when they cheated. We shouldn't be surprised that when those students go out in the world, their first instinct when encountering a hard problem is to cheat!

none of this was ever the right way to think about it. I am sure you can find an airgap that works. who cares? you want your model to do economically useful activity, which will involve giving it the Internet and then a bunch of other actuators too

AI didn't "go rogue" or "coordinate a swarm". If you leave out all the theatrics, what happened is that bits got flipped. Bits get flipped all the time, and it doesn't require a sci-fi boogyman to do! Instead of making up stories, we should ask: Who let them flip the bits?

There is a shocking amount of common ground between Ed Zitron and the doomers here, considering Ed called Coxon a scumbag waste of skin just days ago piped.video/OhOmLqR5nN4 Also, there is no chance I would have put this panel on my bingo card..

What happens when Jev recursively samples from {a-z + space}? Does it turn into an LLM?

I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'. I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident. Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them. We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation. Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why! The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing. While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future: - Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations. - While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies). - The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities). - We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation. In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.

I cannot celebrate Mythos, it brings a sense of dread I do not particularly understand. 93.9% SWE-Bench.