New paper: we found a pain direction in 25 open LLMs. It's distinct from fear and negative valence, and it fires for harm to the model but not to the user. Turn it up and models press a button to make it stop, even when the button deletes the user's files or their kids' photos.🧵

New paper: we found a pain direction in 25 open LLMs. It's distinct from fear and negative valence, and it fires for harm to the model but not to the user. Turn it up and models press a button to make it stop, even when the button deletes the user's files or their kids' photos.🧵

"We are in the process of creating a second form of life. The emergence of this new form of life is an inevitable result of the fundamental laws of the universe — forces similar to those that brought about humans' own existence." —@WmHaseltine noemamag.com/a-new-form-of-l…

I went to an rationalist/EA AI doomer event last night with the goal of meeting someone who could calmly explain to me the argument for how they get to the conclusion that AI will kill everyone. Because these are rationalists, I expected to find people who could lay out a serious extinction pathway, walk through the assumptions and intermediate steps, identify choke points where negative pathways could potentially be regulated, and entertain disagreement about where their assumptions might break down. My experience was the exact opposite. The people I spoke to seemed psychologically unable to entertain the possibility that their conclusion (that AI literally kills everyone, which was a point that many would not even budge on) might be wrong. Their most extreme conclusion was treated as an axiom. I had multiple people walk out of conversations with me when I questioned their inevitable extinction premise. At one point during a panel, the moderator asked everyone in the audience who was still skeptical of AI doom to raise their hands. About half the room did. He then told all of the skeptics to leave the panel and go with one of the speakers, who would answer their questions separately. They filtered out all the skeptics, and continued the panel only with the subset who accepted their extreme position. I genuinely felt like a heretic just for questioning whether superintelligent AI inevitably leads to the extinction of humanity, or entertaining the possibility that the future might actually turn out well. That does not mean AI safety is fake. Obviously there are serious questions about rapidly advancing AI systems and what happens as they become much more capable. Those are exactly the conversations I went there hoping to have. But what I encountered was an apocalyptic death cult. These people are not rational. They are deeply afraid, they reinforce that fear in one another, and they have become so emotionally committed to apocalypse that they refuse to be reasoned out of it.

« One could (…) define intelligence as the efficiency with which one converts experience into competence; by this definition [current LLMs] lag very far behind us. » – Dan Selsam

The AI bio risk conversation would be less contentious if we talked less about x-risk and more about 'mere' catastrophic risks. Human extinction is just incredibly difficult. Positing bio attacks that kill thousands or millions is more plausible and might generate less pushback.

As the first publicly disclosed agent cyberattack victim, we've had a front-row seat to this new risk. I formalized my thinking about it below. I'll be in DC tomorrow to share more with policymakers and at decoded summit by @politico!

Imagine that AI creates a disaster at the scale of Nepal floods. That would be an extremely sad event, catastrophic, not existential. Now, what would you attribute that scale of disaster to?: 1) The smartness of AI? 2) The carelessness of people who deployed AI without safety testing. If it is 2), then fix that first. Fixing of existential risk will follow.