[1/5] The 8 most valuable data points labs should share to help measure RSI: First, RSI would likely accelerate growth in AI capabilities. Thus, companies should report performance on diverse benchmarks for the latest internally deployed models.

My name is Chris Painter, and I'm the President of METR (Model Evaluation and Threat Research). I know we've made a lot of new friends on the internet the last couple of days, so I thought I'd take this chance to re-up what we do and why. Our work is aimed at making sure that if AI really were autonomous, difficult to steer, and close to "going rogue," the public would find out. If evidence exists inside of an AI company that it’s close to losing control of AI, we want to make sure that information gets shared with the rest of the world, including governments and the public outside the company’s walls. This is what we've been focused on since 2022, and over the years we've worked with OpenAI, Anthropic, Google DeepMind, Meta, Amazon, and others on piloting third-party assessments and investigations of this type. We don’t have some private room where we rubber stamp things as “safe” or not. We have had a track record of publishing results on AI that don't cleanly map onto the "doomer" or "accelerationist" labels, and we put in effort to hire people with competing views on AI. We’ve been cited for having found some of the strongest evidence that AI capabilities are improving rapidly (our work measuring AI “time horizons”) while also presenting some of the strongest evidence that, at various points, AI’s capability may be overstated (some might remember our study showing that early 2025 software engineers were actually being slowed when they thought they were being sped up). METR is funded by donations. We don't accept money from frontier AI companies. They haven't paid us for our work, and we don't accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering. Our funding intentionally comes from a wide range of donors, which we’ve shared on our website. Today, when an AI company works with any third-party evaluator or external testing organization (of which there are and should be many), it's entirely voluntary. This often involves NDAs and redactions. To counterbalance this, we have a principle that when we enter into a contract with a company, we try to retain the right to tell the public the terms of the contract we signed, and characterize the nature of redactions that the company chose to make. For example, the report from our independent investigation of the OpenAI-HuggingFace incident included that information. Public disclosure is also a big part of our COI policy (linked on our website). That’s not to say our reports are adequate as oversight. We’re just one organization (among many doing great work), working in a voluntary setup, trying to get good evidence to the public and the world about AI, letting the facts fall where they may.

This a.m., I walked my two youngest kids (ages 7 and 4) to school. I held their little hands as we crossed streets, and gave my youngest a giant hug at his PreK4 classroom door. Then, after walking a block towards home, I opened this app to a terrifying stream of tweets. Of course, I have heard about "alignment" and "AI safety" and "x risk" concerns for years. I took these concerns at face value, especially when I heard them directly from folks at the labs working closest with the technology. And I was grateful so many smart people are working on them. And that's about it. (My work focuses on AI's impact on jobs.) But something has shifted, dramatically, for me in the last few weeks. Between the Hugging Face incident, the damning independent @METR_Evals reports that followed, and the alarming chorus of calls (pleas? shouts? SOS signals?) from inside the labs that we are careening toward potential catastrophe, all of this has made me feel, viscerally, how truly dangerous this moment is. Clearly I am not alone in this light bulb moment. It is one thing for someone in a tropical climate to try and conceptualize snow. It is another thing entirely to be knee deep in it, frozen down to your toes. Somehow hearing the details of the HF incident felt to me more like trudging through snow than theorizing about a possible blizzard; it really hit home. Which is why, standing a block from my children's elementary school in Capitol Hill this morning, I felt sick to my stomach reading my phone. I kept going back to @EvanHub's > 0.1 risk of AI killing us all. TEN PERCENT. *Greater* than ten percent. (To be fair to him, he was clear his worry isn't today's models, but what comes next.) As a parent, it is extraordinary the lengths I will go to keep my children safe against lethal risks with infinitesimal chances of happening. Bike to school? My children *must* wear a helmet. A national study found 2 bicycle-related deaths per million children <16 in states with helmet laws, vs 2.5 per million in states without. So: 0.000002. More than half the states passed a law over that. When I was pregnant, I religiously avoided soft cheese, hyper aware of the risks of listeria, even as @ProfEmilyOster's fabulous books walked me through the actual math. The joint FDA/FSIS risk assessment puts the odds of listeriosis at roughly one in 5 million per serving of soft cheese. 0.0000002. This morning, as I microwaved oatmeal for my kids' breakfast, I made sure I avoided a plastic bowl, to avoid some future cancer risks or something else we'll discover plastic causes. As a parent, I will go to extraordinary lengths to keep my kids' safe. Because that is literally my number one job as a parent. And yet, here I am, outside my kids' school this morning, scrolling through tweet after tweet about the AI arms race, the blistering pace of technological progress, the bee-line for trillion dollar IPOs, and our professed (confessed?) inability to make sure this exceedingly capable technology we are racing to create and unleash into the world won't escape our control and destroy us. Or, more bluntly, kill our children. To be sure, we are tied in knots trying to resolve the game theory dynamics with China, and there are all sorts of competitive pressures and unresolved tensions. But OBVIOUSLY, we need a proactive plan, with real coordination, effective regulation, and an ability to do what my 11-year-old bluntly stated as the clear solution if we are at risk of serious harm: just turn it off. We also need to figure out better ways of talking about this with the American public, outside of the AI-pilled Twitterverse. I just polled my college roommates, sisters, book club, mom friends, none of whom are anywhere close to the AI conversation. Suffice to say, this AI safety conversation has *not* entered the mainstream. Most people I pinged are not following it closely, beyond several listening to @kevinroose on the Daily a few days ago. The newspaper headlines don't speak to them, and they aren't clicking. It seems like a niche AI issue. Others said it all sounded hyperbolic, Y2K all over again. One friend said bluntly, "people get data centers and jobs. They like to get paid and have water." Even those who tuned it to the Daily episode found the issue hard to understand, and overwhelming to wrap their heads around. Terms like "superintelligence" and "alignment" are getting lost on people. Several people told me the paper clip example used everyday language people can actually understand. Another friend said she has never seen a clear explanation of *how* AI could kill us all. Here's my two cents. I think these issues are *exceedingly* important and hugely consequential to literally every person in this country. We need an actual plan, and we need policymakers to step up, asap. Right now, I hear only crickets in DC, where I live. For that to happen, this needs to be a kitchen table conversation, not just an x debate among AI insiders who speak in technical terms that leave normal people behind. This moment demands brilliant communicators -- journalists, experts, activists, concerned citizens, writers, researchers, filmmakers, creatives-- who can relate this to people in ways they can wrap their heads around, who can connect this to what matters in their lives (ahem, keeping their kids alive!), and who can lay out practical steps that policymakers can do to safeguard humanity. In other words, this can't just be talked about on @dwarkesh_sp, it needs to be everywhere, from the View to People Magazine. And it needs to be tackled here, in my town, in Washington, DC. One of the streets I crossed this morning with my kids on the way to school was East Capital. I told my kids to look left, and see if they could spot the capital building. There were too many trees this morning blocking the view.

I appreciate this as an initial step toward more transparent reporting on RSI. But there is still much further data we need to fully understand RSI. In particular, OAI disclosed some evidence about the inference compute usage, number of experiments/researcher, and to a lesser extent, how researchers interact with models of different capabilities (see the green boxes in the diagram). However, the evidence they provided is just on the input side (tokens, lines of code). We urgently need to know how these inputs get turned into algorithmic improvements and [probably internally deployed] AI capabilities. Even just a simple time series on those outcome variables would be extremely valuable. You can then know, as AI helps researchers do more AI R&D, how the corresponding algorithm efficiency grows (the blue arrow C -> A dot). Right now we are missing this key puzzle piece.

We develop a mechanism design framework for AI alignment and control: arxiv.org/abs/2609.01595 It’s largely conceptual but we offer stylized applications to failure modes (sandbagging, alignment faking), safety practice (scalable oversight, peer prediction), and a way to think about the value of alignment, interpretability, capability, and control.

Extremely happy that Elasticity has been getting advice & help from @yelizarovanna and Olivia Benoit.

We’re hiring in SF! Encode cut its teeth in DC. People said AI politics was a bad bet; we made unlikely friends, won the fight to keep states in play, and helped pass the Big 3 of AI laws (CA SB 53, NY RAISE, IL SB 315). It’s time to double up in SF. Pacing the Frontier (pacingthefrontier.com), which we helped organize, was recently signed by 1,378 frontier lab employees and endorsed by both OpenAI and Anthropic. The last time we felt this rush of opportunity was Sacramento in 2024. AI risk has gone from an online culture war to a fact of life. SF is full of people who want to act, lab employees especially, but don’t see a place to plug in. We need someone to help us build the grid. Encode’s lab engagement has been running on adrenaline and personal relationships. That got @_NathanCalvin and me here, but it’s time to level up. Misalignment is getting scary. “Pacing” has real legs now. Researchers are using their voices to influence lab leaders and alert the world. Many of those researchers say that soon, their jobs - and the voice their jobs give them - will be unrecognizable or won’t exist. We want to hire someone who is: (1) Mission-aligned. You want to ensure AI doesn’t cause catastrophe (or kill everyone) and the future is the best it can be (2) Extremely organized, high-functioning, excited to add structure so ad hoc actions compound (3) Pragmatic, able to pivot when crazy things happen (if the last few months are any indication...) (4) Hard to typecast - not because you lack beliefs, but because you see the best in many kinds of people. The role will start with supporting lab engagement and expand. It’ll be insanely fun (we take the work seriously, not ourselves). You’ll visit DC regularly as AI’s political salience takes off and get to know a team that feels like family. If you’re interested, email me by September 4 at sneha@encodeai.org with the subject line “SF Special Projects” and a few sentences each on (1) a relevant project you’ve owned and (2) what you think Encode SF should do next. If someone you know could be a good fit... I want to meet them!

“Dario believes frontier AI is too powerful to distribute; we believe it is too powerful to centralize.” The problem is that both are true.