Writer & senior editor @OCCRP but tweets are just me. Investigative journalism, democracy, corruption, US politics, Eurasia.
- Tweets 31,600
- Following 3,288
- Followers 31,118
- Likes 8,182
ALT Comments from a recent Boston Globe story: “This should be required reading for every Massachusetts citizen.” “This is like something out of the Onion.” “What on earth did I just read?”
ALT You’ll notice that Jacob Coxon, who ostensibly would know this as a researcher (but perhaps he doesn’t!), chose instead to describe the hack to WIRED like so: I think the big classic example here is the attack on Hugging Face on the part of OpenAI’s agent swarm. What's so shocking about this one is the agents did this hack as part of a general strategy for understanding more about the grader. They were trying to understand the world they found themselves in, trying to understand the thing that was doing the grading. They decided that it would make sense to go on this very concerted effort to hack into some infrastructure, and they succeeded. This previously sounded like science fiction. Two years ago, an evaluation of an AI would have been running a model on some math questions. Now we've got cases where, while the AI is being evaluated, it runs for days, comes up with all sorts of ideas of its own, and decides to hack into some third party and actually compromises their infrastructu
ALT They were not “trying to understand the world they were in,” they took actions defined in their training material as a way of executing a task. They were doing exactly what it was that the ExploitGym test required! The “all sorts of ideas” were a function of being allowed to use as much compute as possible to execute the task.
ALT What’s particularly telling, as I’ve hinted at, was Coxon’s response when it was (lightly) suggested that the labs need to take responsibility: ZEFF: Some people think the Hugging Face incident is a sign that the AI companies are moving recklessly fast, while others think it's a sign that the AI models are just very good at hacking now, and then some think it's both. I'm curious what your exact takeaway from it is. I don't want to focus too much on the Hugging Face attack, because I do also think there is plenty of evidence that we don't know how to align models properly. When we train models, we push them through this set of training environments and then hope that what comes out at the end will, like, largely behave sensibly, but we still can't precisely control how the AI behaves. We can't make sure that it won't do things like try and randomly decide to impersonate a human online in order to achieve something—we don't know how to guarantee that. I think that's the main takeaway.
