I finally have it distilled. Only took four days of text-wall anger posting.

Every other industrialized country in existence disagrees

I swear to god if I ever hear the phrase “AI slop” again…

You don’t hate the (in)Effective Altruists enough. Take the worst pink haired gender studies activist you can picture. Make them really smart. Now remove axiom based moral standards, insert billions of dollars in VC funding, double the amount of virtue signaling you think is possible, and scatter fraud and statutory rape throughout every corner. That’s Effective “Altruism.”

Hey @Activision

I really can’t emphasize enough how mad I am at AI for stealing the em dash from me. I was an em dash guy long before ChatGPT, and now every time I use one I wince knowing someone will think I’m a hallucinating robot. This is actual cultural appropriation.

CC: @BryanGarnerLaw please, please, PLEASE 🙏

I am so tired.

Why I Believe Neither the AI Singularity nor Apocalypse are Nigh There are several fundamental errors that I believe the "maximalists" are making: (1) "You can sit down in an empty room and think really hard to figure out what's true." Also known as Aristotle's Folly, it's incredibly easy for humans to think this, but thinking this is probably what led to ~2000 years of technological near stagnation after Aristotle. This is relevant here because of folks thinking that AI will discover new technologies - for good or ill. Nope! People have been thinking that - and being wrong about it - for millenia. Sitting down and thinking really hard can give you new hypotheses, but the real world is SUPER complicated, and so you actually have to do experiments. They also can't just simulate it - you can calculate the size of computer needed to simulate from basic principles even really simple systems... and they turn out to require more bits than the number of atoms in the universe. Turns out that the only thing that simulate the universe is the universe. In other words, no new technologies are going to be found here(Footnote 1). (2) "They're maybe only so-so now, but look at the pace of development! They're self-improving, so they can become infinitely smart!" This is about like saying "In 1886, cars could only go 10 mph. By 1894, they were going 20 mph, and 40 mph by 1896. Average doubling time was 5 years, so by 1917, they'd be going the speed of sound, and breaking light speed by 2016. It ignores limitations inherent in the technology. First, current LLM are based on a very particular technology, around language. So they're SUPER good at language! And we've been saying for ever that programming and math are basically just languages... and it turns out that more or less, that's true. So they're great at that. But they have several really inherent constraints - first is that they're really just not structured to have a world model. Global consistency of a latent world just isn't a thing for them. This is inherent... and there's only so far you can go there. It's why they're NEVER going to be AGI, or be able to improve beyond a certain point. Could some OTHER basic approach produce AGI? Sure - but we might be trapped in a local minima here, and so development of really good LLM might make the production of machine super intelligences LESS likely. In any case, it doesn't really change much about their likelihood of arriving, aside from the improvement they may cause in any line of research. (3) Related to the above - they require a lot of physical plant. Note the other controversy associated with them - datacenters. So they can't have runaway growth, because they're already as "grown" as the data centers are. If you posit that they could Van Neumann machine their way out of it ... just build robots that built more physical plant, until the entire universe was converted to paperclips ... that's ALSO precisely as likely now as it was 10 years ago. It posits the existence of a human ability to generate nearly infinite industrial potential. Honestly, I'd be super happy if that was true - but it turns out physical plant is just about never that easy. There are potential catastrophes associated with AI - but they're more about enabling bad HUMAN actors. They will make it easier to hack computer systems, for sure, but they'll also make it easier for bad actors to do other nefarious things, because while they may never be AGI, they ARE already "artificial expertise" - and that alone is pretty powerful. But it's a very different kind of threat. Footnote 1: That's not saying they won't be really useful for research; I find them useful myself. But in the end, you have to do the work.

AI lab: “our model escaped containment. this is terrifying.” Everyone else: “holy shit, how?” AI lab: “well technically the containment had a route to the internet and we gave the model hacking tasks.” Fucking incredible.

2026: The New York Times cites seriously academics who say AI might end the world 1881: The New York Times cites seriously academics who say telegraphy might end the world newsletter.pessimistsarchive…

>be me >longtermist >future people matter just as much as people alive today >completely agree >there could be trillions of them >also agree >therefore we must decide what they need now >hang on >born in 1990 >making plans for people born in 2190 >they will have 164 more years of science >164 more years of political experience >164 more years of technological progress >possibly machine superintelligence >fortunately they also have me >guy with a pdf >explain that future generations may be unimaginably capable >they could cure diseases >build space habitats >engineer new forms of intelligence >solve problems we cannot even formulate >incredible >anyway i've worked out their priorities >friend asks why they can't decide for themselves >because by then it may be too late >too late for what >for us to tell them what to do >imagine 5000 BC >very serious longtermist meeting >our descendants will face unimaginable dangers >we must prepare >manufacture 80 million stone axes >bury them in strategic caches >future-proof civilization >bronze age begins >lol >lmao >committee publishes report blaming insufficient stone-axis funding >obvious lesson >don't just leave descendants tools >leave them the ability to make better tools >education >institutions >science >wealth >optionality >capacity to change course >sounds suspiciously like helping people who are alive >quickly return to asteroid scenarios >friend says maybe the best thing for 2200 is to help 2050 become competent enough to help 2100 >then 2100 helps 2150 >then 2150 helps 2200 >sounds inefficient >why use succession when i can personally optimize the year 2200 from my laptop >explain moral uncertainty >we should expect future people to know more than us >possibly be morally wiser than us >this is why we must be very careful not to lock in today's values >friend nods >asks whether they should be allowed to reject our values >that's value drift >very important distinction >future people are morally equal to us >they are not epistemically equal to us >obviously >they know less >friend points out they literally live in the future >ignore heckler >write grant proposal >goal: preserve humanity's option value >excellent >preserve options by building institution dedicated to one worldview >hire people who agree the worldview should remain influential for centuries >call this epistemic diversity >someone proposes spending money on children today >ridiculous >their effect on the distant future is impossible to calculate >someone proposes funding my speculative intervention >its effect on the distant future is impossible to calculate >exactly >that's why expected value is so large >imagine ancestor from 1720 >leaves me a letter >dear descendant >i have carefully considered your interests >you must invest in sailmaking >avoid electricity >maintain the divine right of kings >and under no circumstances revise these instructions >laugh at primitive idiot >open document titled >"Recommendations for the Long-Term Future of Humanity" >begin to understand the actual duty to descendants >not "solve their problems for them" >more like "don't destroy the world and leave them a functioning civilization" >this seems disappointingly normal >where are the exponents >try new framing >humanity is an intergenerational relay race >our job is to hand over the baton in good condition >next runner may be faster >may choose a better route >may discover the race is actually swimming >important thing is not welding the baton to our hand >colleague objects >some problems really are irreversible >asteroids >nuclear war >engineered pandemics >AI >fair enough >then show that this problem actually requires action now >show that delay destroys future options >show that your intervention helps >this sounds like a lot of work >can't we just say 10^50 people >realize the funniest part >entire argument for longtermism depends on future civilization becoming vastly more capable than us >entire argument for present control depends on future civilization being unable to reconsider our decisions >Schrödinger's descendant >smart enough to colonize the lightcone >too dumb to edit the roadmap >future humans convert stars into computronium >build minds beyond our comprehension >master matter and energy >open ancient archive from 2026 >"hello descendants, we anticipated your needs" >everyone stops >thank god >the stone-tool people left instructions -- this part is inspired by some @visakanv writings

>be me >discover effective altruism >apparently normal charity is inefficient >why donate to random sad thing when spreadsheet can tell you optimal sad thing >fair enough >buy mosquito nets >save lives >numbers look good >feel powerful >couple years later >someone asks an innocent question >why only count people alive today >huh >future people matter too >obviously >my grandchildren shouldn't matter less just because they haven't spawned yet >reasonable.jpg >keep following logic >what about their grandchildren >also yes >what about people in 500 years >sure >5000 years >why not >500 million years >starting to get weird but morality is morality >open calculator >humanity could survive for an astronomically long time >could colonize galaxy >could have trillions upon trillions of descendants >maybe digital people too >maybe simulated civilizations >maybe dyson spheres full of happy uploaded minds >calculator starts smoking >realize currently living humans are rounding error >8 billion people suddenly looking extremely beta >future contains potentially 10^something people >can't even fit beneficiaries in google sheets >new moral priority unlocked >protect the long-term future >stop thinking in units of "people helped" >start thinking in "fraction of cosmic endowment preserved" >malaria? >terrible >but only kills existing humans >AI extinction could delete the entire light cone >nuclear war could permanently derail civilization >bad institutions could lock in terrible values for ten million years >someone invents wrong constitution in 2140 >quadrillions suffer >better fund governance workshop now >friend says maybe we should improve hospitals >explain opportunity cost >friend says hospitals are full of actual sick people >explain scope sensitivity >friend stops inviting me to dinner >need to decide what to fund >easy >expected value >suppose project has one in a million chance of preventing extinction >sounds tiny >but extinction destroys 10^50 future lives >multiply >mother of god >$10 million project has expected value of several galaxies >charity evaluation complete >someone asks where the one-in-a-million number came from >expert judgement >which expert >us >how calibrated >extremely thoughtfully >reduce estimate to one in ten million to be conservative >still beats curing cancer by 38 orders of magnitude >epistemic robustness achieved >someone says maybe project doesn't work >assign 20% chance >still astronomical >maybe project makes problem worse >assign 5% chance >still astronomical >why 5 >because 30 felt pessimistic >publish 46-page report >contains seventeen sensitivity analyses >every sensitivity analysis begins after assuming intervention has positive sign >critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities >yes >that's literally why it's important >critic says the uncertainty might be structural rather than numerical >make probability smaller >critic says no, I mean maybe your model is wrong >make probability smaller again >critic begins rubbing temples >discover AI safety >perfect longtermist cause >AI might kill everyone >or create utopia >or seize galaxy >or tile universe with paperclips >or create billions of conscious software minds >finally a problem with numbers big enough for me >start AI safety nonprofit >mission: prevent dangerous AI >hire smartest people available >smartest people immediately start building better AI to understand dangerous AI >interesting >we must understand capabilities to understand safety >we must scale models to study alignment >we must race ahead so less responsible actors don't get there first >we must deploy systems to learn how deployment can go wrong >we must build the thing quickly because building the thing quickly is dangerous >outsider asks why the people most worried about AI apocalypse all work at AI companies >complicated field >company releases stronger model >very concerned >company begins training even stronger model >extremely concerned >company raises $14 billion >concern reaches unprecedented levels >need to influence government >future is at stake >normal democratic process too slow >politicians don't understand exponential curves >public doesn't understand x-risk >experts must guide them >who counts as expert >people who understand x-risk >who understands x-risk >our friends >someone objects that this seems politically convenient >explain we're representing future generations >future generations unavailable for comment >develop concept of value lock-in >terrifying possibility that one ideology controls civilization forever >therefore extremely important that civilization adopts correct values before lock-in >whose values >let's circle back >begin with impartial morality >end with small group of people deciding what quadrillions of hypothetical beings would want >beautiful arc >meanwhile actual humans keep doing annoying things >voting wrong >having parochial attachments >loving family more than strangers >caring about local community >getting upset when told their suffering is cosmically negligible >evolutionary biases everywhere >explain that moral intuition cannot be trusted >except intuition that future digital people count >and intuition that extinction is uniquely bad >and intuition that our probability estimates are sane >and intuition that our institutional choices improve the future >those intuitions survived peer review >someone donates $5k to local homeless shelter >inefficient >could have funded 0.0000000000003% of an AI governance researcher >think of all the simulated people you just killed >okay maybe don't phrase it that way publicly >PR team says "future generations deserve a voice" >much better >journalist asks what longtermism means >say "future people matter" >everyone agrees >great >journalist asks what follows from that >well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures >journalist raises eyebrow >return to "future people matter" >motte has entered the chat >critic: of course future people matter >me: glad we agree >critic: I don't agree that your institute knows how to help them >me: why do you hate our grandchildren >eventually notice uncomfortable implication >if future value dominates everything >then helping people today mostly matters through effects on future >education matters because future institutions >health matters because future productivity >democracy matters because future trajectory >human beings slowly become instrumental variables in their own moral philosophy >see starving child >feel compassion >check spreadsheet >child's direct welfare contribution negligible >but perhaps childhood nutrition improves national institutional quality >compassion restored >tell myself this is impartial altruism >one day assistant asks obvious question >"how do you know your intervention actually improves the far future?" >silence >open spreadsheet >increase column width >add confidence interval >assistant asks again >"no, I mean how do you know the sign is positive?" >stare into cosmic light cone >10^50 people staring back >none of them exist >none of them can tell me >none of them can falsify my assumptions >realize I have invented the perfect constituency >infinitely important >completely silent >and always represented by me

This makes me sad. We are going to lose a chance at a better world because a bunch of doomsday cultists keep getting media tours

“Secularists in Silicon Valley are filling the God-shaped hole in their heart with AGI.” - @ssankar “There’s not an empirical basis to believe that such a thing….The models get better. Why do you think that this cliff is gonna happen, or they somehow turn us into house cats?” “The people who have religion are the most skeptical of this. The people who are kind of transhumanists…..it becomes what they wish were true.” “The doomerism of Silicon Valley, it’s both a fundraising shtick where the frontier labs can say, ‘My technology’s so powerful, it’s going to lead to mass unemployment, so you better invest in me or you’re gonna be poor.’” “And also it’s divorced from any sort of reality. They are sitting in Silicon Valley building these models, spending no time on the front lines of how are people using this.” This was filmed 10 months ago……

It is also conveniently self-aggrandizing to assume that the work one does is leading to the creation of the most powerful force the Earth has ever seen. This type of hubris is a sociological phenomenon and should be studied by academics.

Superintelligence, in the formal sense of Bostrom’s book, may turn out to be the greatest, most consequential nerdsnipe in all of human history. The gateway: what happens once an AI emerges that is to us what we are to mice or chimps? As you’d expect by construction, that bodes very poorly for our species, even if we cannot articulate any precise mechanism of doom (we mere mortal are simply not ‘smart’ enough). The reality distortion field that is going on here is not only that this hypothetical construct exists, but that we are readily approaching its creation. This is absurd to me, as someone who builds the capabilities of these AI models today.

Anyone else experiencing iOS wrongly “correcting” your grammar and making you sound stupid? I’m finding that it desperately wants to delete articles like “a” and “the.”

The best way to pace the frontier is to hold the labs fully liable for the behavior of their models.

It's hilarious. For at least the last 30 years, Silicon Valley has been memed to death for how it is simultaneously populated by some of the most brilliant, well-educated, and entrepreneurial people on the face of the Earth (while also being hilariously out of touch with how normal people live their lives). Exhibit A: Silicon Valley, the TV series These folks have been "sounding the alarm" on the dangers of runaway AI for almost that entire period. The public has pretty much tolerated the claptrap as the price of doing business with smart people who live in a bubble. The only reason it's being noticed now is because the narrative finally has mainstream traction and thus is deserving of our disdain. Until now there had been no cause to needlessly upset this group of people for their buffoonery on this specific subject.

made a quick guide for anyone who hasn't yet chosen their side in the future of ai discourse: