A recent conversation about "how hard is alignment?" had an interesting bit where: 1. we'd narrowed the topic to "will Constitutional AI scale to unbounded superintelligence?" 2. They thought "70% yes" 3. I was arguing "we have enough information that the current best guess should be >50% no." And they felt like I was expressing more confidence than them (and, felt confused about where my confidence was coming from). I actually kinda get where they're coming from but this is fairly interesting and I'm not sure what's going on. (something like, it's okay to be confident... but weird to be metaconfidently confident?") Previously, people would say "How can MIRI people be confident enough to write a book title like "If Anyone Builds It Everyone Dies." ASI has never been seen before, this is a highly specific claim. I got where those people were coming from. The MIRI people do seem very confident and it's a pretty reasonable epistemic state to find that weird. But now it seems to me like distrust/confusion in metaconfidence runs deeper and even extends to "highly confident that the correct confidence is >50%". And this feels like some important puzzle piece I don't quite understand. I'm curious if anyone reading this feels weirded out by that, and is up for sharing more details.

My basic argument for why it's reasonable to be "confidently >50%" is: "For a given set of knowledge, it's just practical to evaluate the arguments for, evaluate the arguments against, and see that there are more/stronger argument for than against at present time." You can still believe that you might totally update if you learn more. If you've just learned 5 facts about a thing that points one way, and suspect that there are 20 more facts you'll learn soon, you'd certainly hold "probably true" lightly and not take any crazy actions based on it.

It's interesting seeing the "openly advocating for loyalty over honesty" discourse. I don't think know any directly in my filter-bubble who'll care what I say, but, here's me attempting to bridge/dialogue: I see two major principles: 1. All-else-equal, it's better to be a good trade partner, who people don't regret interacting with. 2. All-else-equal, we want information about poweful institutions being incompetent or corrupt or mistaken, to be accessible for public sensemaking. I'm guessing the "loyalty advocates" aren't oriented around "good epistemics are among the most important things to invest in." People understanding problems is what allows them to fix them. There's some awkward disagreements within the rationalist-EA spectrum, which doesn't exactly map to loyalty-vs-unfitlered-outspokenness, but is like "do people implicitly try to protect each other's reputation?". @ohabryka call these "mutual reputation alliances." This results in bad decisions getting under-discussed and propagated longer than they should have. Here's a brief summary of the positive vision that squares the two principles. A. Be upfront. If you'll predictably dislike everything an org is doing and want to publicly criticize them all the time, be upfront about that. This may result in the org not hiring you. They might hire you anyway, if you are very competent. It's damaging to the social fabric for institutions to assume they have to ferret out adversarial spies. Sometimes it's worth being an adversarial spy (i.e. I'm glad there were adversarial spies in Nazi Germany, as the obvious example). But it's easy to think you're in more of an all-out-war than you are. B. Be fair, and respectful. Just because you're going to criticize people doesn't mean you have to be an asshole about. Don't downplay flaws, but, don't exaggerate them either. Business-as-usual-political-discourse often involves digging into every single bad-looking thing your opponents do and making them look bad. Don't do that. Be proportionate. B2: It's possible for a culture to decide that it _is_ a normal, respectful thing, to criticize things. In business-as-usual-politics, criticizing people is seen as an attack. You have to actually put a lot of culture-setting work it to make it _not_ be (automatically) an attack. This requires several moving pieces to align at the same time. You need a norm of not pouncing-on-people for every damn thing. You need a norm of not seizing people's "changing their mind" as a quick political gotcha. We don't automatically live in this world. But, you can choose to be the change that makes the honest, high-integrity world more real. ... I suppose I need to address the elephant in the room: The Trump administration seems both unusually corrupt and unusually loyalty focused. l think this is bad. I also think it's quite likely that me and my allies (i.e. AI safety advocates) need to be having positive-sum relationship with the Trump admin. This is tricky. I'll say upfront: I think it is very important that the government culture of Trump ends in 2028 and many elements of it reverse. But, I also think it'd be unironically great if Trump signed a deal with China banning uncontrolled AI takeoff, and got credit as the guy who led the most important deal of the 21st century. I meanwhile think many people are critical of Trump in a disproportionate way that doesn't track reality, failing to distinguish between his most important wrongdoings and pretty minor stuff. And many democrats seem wrong about some worldview assumptions (i.e capitalism is good actually, de-regulation can often be good, etc). Getting into all the details here is beyond scope. But: I unironically support trying to positive sum trade partners while also being, in some domains, enemies trying to stop each other from doing things-that-seem-bad. I don't see this as a contradiction. But it does require a worldview adjustment that is non-obvious to many people.