essays

But is it really spurious :-D ? I do agree with Sonnet, used to drink more green teas in evaluations and do drink more oolongs now

defining it prescisely is difficult, but i think there's something there worth being curious about! when i write a big program, one nice thing i can do is look at the code and notice bugs and fix them. i can also ofc use the program normally and notice bugs. but the former is often more efficient for finding certain kinds of bugs, just because randomly bumping into the bug might be really hard. even if I am learning from my experience of where i previously found bugs to try and develop a mental model of ways the code might be flawed (i.e RL), my life would still be a lot easier if I've already seen the code. the thing with NNs is that looking at the weights doesn't help you over the naive RL baseline in the same way as looking at the code of even a very large complex software project. this should be kind of surprising if we thought looking at the weights + algo gave us complete understanding! also, there are (admittedly unprincipled, hacky) transformations of weights/acts that give us a better mental model and therefore slightly more ability to find bugs in the models. even if you don't want to call this thing understanding, it should be clear that there is a thing that is missing that helps our ability to make systems more robust (there's one slight wrinkle which is because NNs are differentiable, you can do gradient based attacks, which destroys RL for attacking image models. but they don't work very well on transformer LMs, even if you do some tricks like GCG to get around the discreteness of the input space, so that in practice RL is SOTA for automatic red teaming.)

i mean eventually we all will

We've written a post arguing that latent reasoning architectures (aka 'neuralese') would substantially increase misalignment risk via making oversight much harder. In the extreme, we could see massive 'neuralese hivemind swarms' where the agents think and communicate in latents, likely making oversight nearly entirely reliant on observing the actions these agents take. (And these agents would have huge amounts of time to reason about obfuscating their actions if they wanted to do so...) Individual agents doing extensive latent reasoning would also be concerning; in the post we discuss how above some threshold of latent reasoning, agents may be able to perform difficult-to-detect and reliable steganography for communication and further reasoning. We argue both that latent reasoning architectures would make chain-of-thought no longer very useful for oversight (by eliminating or greatly reducing the need for verbalized reasoning) and that, without these architectures, it's likely the value of chain-of-thought for oversight could be preserved. redwoodresearch.org/blog/lat…

So, we have a couple reasons to expect to be getting emotional representation related to the assistant response, even under the hypothesis that the pain axis works roughly like Anthropic's emotion concepts do: that is, not picking out a ‘self’ representation but rather a ‘current speaker’ representation. And I think this is plausible, given the similar experimental design.

Completely insane. Do dogs not feel or want things? There is such a deeply anti-intellectual refusal to even consider that philosophy of mind is complicated across the culture right now.

Restatement (Second) of Torts §§ 506 (Am. Law Inst. 1977) - viz. the wild nature of animals that they keep