prediction: within 2 years, we'll see a large cyberattack powered by a swarm of AI agents
it'll be a 9/11-level wake-up call with devastating real-world consequences, and this will force people to take present-day AI risks more seriously, even before existential risks take center stage
what could this look like?
• a major bank's records get wiped out
• a large power grid taken down
• an entire fleet of airplanes grounded
• millions of android devices getting hacked
• a sudden, inexplicable stock market crash
• all US AI providers going down simultaneously 🤔
• ...
why do I believe this is coming?
because AI gives offense an **asymmetric advantage**: attackers can scale cheap, persistent attempts faster than we can possibly can secure every target
⇒ think of agent swarms like millions of cheap autonomous drones, but for cyberattacks
when deployed at scale, some of these drones are bound to get through, even if you have the best defenses in the world (as we're learning in ukraine and iran)
the most urgent risk is from malicious actors like terrorists and lone hackers: sophisticated, sustained attacks that previously required resources they simply didn't have
the gap between state-sponsored cyber attacks and what lone wolf hackers can pull off is shrinking rapidly
but malicious use isn't the only risk; the OpenAI-HF incident showed swarms of misaligned agents coordinating unauthorized attacks. rogue swarms and loss of control deserve attention too
but their consequences so far have been limited compared with what i'm worried about here. my immediate concern is people deliberately trying to cause harm, which is known as malicious use (
cultural-alignment.com/risk-…)
**the frontier labs are aware of this risk**
but they can only do so much, and critically, they're not incentivized to spell out the urgency of these dangers quite so publicly & explicitly
openai's project daybreak and anthropic's project glasswing, for instance, are amazing initiatives, and i'm genuinely glad they exist
but i struggle to see how they'll cast a net wide enough, fast enough
these initiatives are mainly aimed at providing select critical infra and enterprise customers frontier cyber defenses ahead of the oncoming deluge of offensive cyber attacks, which is a very worthwhile endeavor
though i struggle to see how this will be enough
there's simply no way that every critical company, government agency, and the billions of consumer IoT devices around the world will move fast enough.. at least not until one or more devastating agentic cyberattacks force the issue
and then we'll be right back to "only a good guy with AI can stop a bad guy with AI"
which i guess is the best we can hope for until shoggoth comes to save us from ourselves...
nitter.net/transitive_bs/status/2…
OpenAI's board finally realizes what Ilya saw