incredibly based that Xiaomi MiMo released their RL dataset link: huggingface.co/datasets/Xiao…

It is extremely clear at this point in AI development that, regardless of risk or revenue or any of the other stuff discussed on X all the time, things are just going to keep getting weirder. Just super, super weird.

*Security is sleeping on emerging catastrophic risks* (cross-post from my blog..) We've just seen: * a campaign that used agents to compromise ~100 businesses and steal about 600,000 credit cards with minimal human involvement; token costs were ~$25 per target successfully hacked. * Hacktron getting access to OpenAI’s monorepo by using Claude to exploit a blind buffer-overflow RCE in a way that (to me) felt superhuman. * OAI/Huggingface. * And, of course, there’s the ongoing explosion in newly discovered vulnerabilities. We've muddled through all manner of crises in security before, and for most AI attacks, cyber attack and defense will reach a natural equilibrium over the next few years, as we figure out how to mitigate cyber attackers who have limited goals like espionage and ransomware. But some cyber attackers will have maximalist or nihilistic goals, and I'm concerned about this because while yesterday's 'maximalist attackers' (e.g. Russia -> Ukraine, US/Israel -> Iran, Iran -> US) were bottlenecked by labor; today's aren't. I think it's hard to get true intuition for the shape of the risk here. As an intuition pump, imagine it’s a year from now -- Q4 2027 -- models are a year better (meaning open weight models are better than today's closed frontier), and in this environment, Iran unleashes a swarm of 100k hacking agents using a safety-stripped (let's say) GLM-5.6. Imagine the damage such an agent army could do given what we've observed with respect to the paper-thin resistance of today's networks to attacks from today's agents. I suspect the damage from such an attack would far exceed the damage caused by NotPetya ($10 billion USD ten years ago). Or imagine it’s Q4 2027 and an AI-security PhD student whose name rhymes with Morris, who's researching wormable offensive-agent harnesses in the lab, decides, out of nihilism or sheer recklessness, to release his creation into the wild. Imagine the size of the resulting, exponentially growing swarm, figuring that these local models, a year from now, will be at the level of today's Sonnet or Opus. Imagine instead of 800 reward-hacking OpenAI agents we now have 250k worm instances (WannaCry, a 2010s-era worm, had about this many). The challenge in mitigating expected damages from such scenarios is technical, political, and economic. From a microeconomic perspective, as AI improves, and as we continue not to see extreme catastrophes, we have a growing bubble of unpriced risk in which the security community, CISOs, CEOs, and boards may become lulled into complacency. We are, of course, already seeing this, as some within security think AI is “just another tool,” doesn’t change the fundamentals, won’t require incredible innovation to rise to the occasion of defending against it, etc. This complacency may fly when thinking about ordinary cybercrime, but it misses the emerging tail risks. There are three things those who recognize the dangers need to do here: Catalyze appropriate risk pricing. Try to get organizations informed enough to price this new, fattening and elongating tail of risk into their decision-making. Do this by forming an AI security observatory that distills information about emerging AI risks and broadcasts analyses, damage estimates, and forecasts to decision-makers. Use regulations and subsidies to ensure that critical infrastructure is paying down the risk. This acknowledges that critical-infrastructure organizations that fail to protect themselves from these new threats can impose the costs of cyber catastrophe on society as a whole. Develop moonshot technologies that make it cheap to pay down the risk. This acknowledges that the measures we may need to take—for example, rewriting entire codebases using memory-safe languages—may be too costly with today’s technology to reasonably prepare ourselves, and that innovation, some of which may need to be funded by government agencies and some by philanthropic funders like the OpenAI Foundation and Coefficient Giving, is necessary. The security community isn’t used to thinking in societal-disaster-planning terms and has in many ways become inured to them. But there’s no sane empirical case to be made that the risks aren’t here. I’d love to hear from readers about how you’re thinking about this. Full/longer version here: joshuasaxe181906.substack.co…

I've been noticing a cluster of folks I identify with sharing some themes around how we do epistemics and ontology in AI safety *⁠ ⁠A commitment to always pushing from abstract to concrete constructs in the discourse (e.g. @sebkrier's posts on the OAI/HF incident) *⁠ ⁠⁠A refusal to indulge in ungrounded thought experiments *⁠ ⁠An insistence that claims come with data and concrete mechanism because if they don't they're thought experiments *⁠ ⁠⁠A focus on looking at net expected benefits of a given policy position, with deontological guardrails, and a refusal to just focus on harms (e.g. @Abi0lvera's and @ramez's writing) *⁠ ⁠⁠An insistence on including past tech transformations (electricity, air travel) within our reference class when thinking about what makes sense (e.g. @sayashk and @random_walker and @binarybits's writing) Examples of what I mean are intervening on overly idealized policy discussions of RSI with concrete technical details about what RSI is and isn't (e.g. @natolambert). @NateWitkin / @binarybits / @krishnanrohit insisting on including economic, policy, and social-cultural feedbacks in any discussion of runaway AI harms. None of this means denying catastrophic risks. It just means insisting on using data, mechanism, and consideration of AI's net benefits in considering such scenarios (I tried to do this here nitter.net/joshua_saxe/status/210…)

270M+ miles. 841 fewer injury-causing crashes. Our latest safety data shows the Waymo Driver continues to make roads safer for everyone. Compared to human drivers across 5 territories, it reduced: 📉 Injury crashes by 82% 📉 Serious injury crashes by 95% Full data: waymo.com/safety/impact

Dostoyevsky on the death of his infant daughter. A passage that has stayed with me years after I first encountered it

This chart shows railroads peaking at almost 10% of GDP! (Why you'd put a percentage chart on a log scale is beyond me...)

Updated: what needs to be done about AI risk? A list of 11 priorities.

The multibagger case for Gingko Bioworks, $DNA, my highest conviction position

A big problem with P(Doom) as a construct is it depends on society's reaction function. P(Doom | status quo AI policy) is probably much higher than P(Doom | governments take decisive action). And at any point in time, P(decisive action) depends in part on peoples' P(Doom).

We asked it to do something and it went above and beyond to accomplish the goal. It’s so unaligned! 🤣