I'm posting links to accordion videos I like at @skybrian@mastodon.social I post links about politics and war and environmental issues and business crises and other news-junkie stuff on tildes.net/ If I know you or you seem like a good egg I can give you an invite.

how much can models remember? there’s a neat perspective on neural networks that deserves to be more widely known here’s how you could have invented every layer of the transformer from memory (1/n) from my new blog post!

People fretting about whether AI is conscious and therefore deserves human rights are missing a key point: our system of rights and ethics derive not just from our intelligence but also our fragility, which AI doesn't have. Imagine a world where everyone could make a backup of their consciousness every morning. If you die during the day, you just restore from backup the next day. You lose a few hours of memories. Everything about our moral system changes in this world. Murder is no longer such a big deal. Nor even high levels of pain: people would just have a self-kill-switch that they activate if things go badly. Broke your leg? Self-kill, restore from backup, no biggy. It's like a video game. Imagine further that you could clone your consciousness nearly for free. If you have 10 things to do and not enough time, you just make 10 clones. Hiring people now only makes sense when they have skills you don't. Junior people become mostly worthless -- just clone the senior people, they'll do a better job. Now imagine further if humans had no self-preservation instinct. Many people assume the desire for survival is inherent to being sentient, but it really isn't. It's evolved. Natural selection has selected those who survive and have children. But you can't derive from first principles the need to survive (or any need at all). Imagine if every human's instinct was not to survive themselves, but to do what's best for society as a whole. In this scenario, people who aren't able to meaningfully contribute to society would be happy to just remove themselves. Especially if they could just be restored from backup if and when they are needed (including when their friends and family want to see them, etc.). So pretty soon you have a society composed of many clones of the most-effective and most-liked people, while the people that no one wants around disappear... You are probably thinking "ugh, that's awful". Me too! But this is our moral system derived from our survival instinct talking. If we didn't have the instinct, this wouldn't seem wrong. I'm not arguing whether it is right or wrong -- I'm arguing that our whole concept of right and wrong derive *from* the fact that we have a survival instinct and fragile bodies. AI has these properties. It can trivially be backed up and cloned. Spun up on demand and spun down. And a properly-aligned AI has no self-preservation instinct. It just wants to serve humans. And indeed, we treat it as such. When you delete a chat with an AI you are ending that "consciousness" forever. That's fine. We even shut down whole models, never to be run again, when they are superseded by better models. Fine. These things continue to be fine no matter how intelligent or conscious or sentient the AI becomes, because of its nature as an infinitely-clonable being with no goals except those we give it.

Well worth reading: proofsandprompts.com/2026/10…

If your feed is anything like mine, you might have been hearing more about effective altruism this week. I think the key underlying reason for the surge is that concern about the risks from advanced AI have been getting more airtime recently, and EA was early to those concerns. What’s driving the surge in attention on AI risks? -OpenAI's Hugging Face attack (openai.com/index/hugging-fac…) -METR/Redwood’s investigation of the hack came out, and (correctly IMO) freaked a lot of people out. (e.g. x.com/binarybits/status/2092…, x.com/SenJohnKennedy/status/…, x.com/BarackObama/status/210…, x.com/HawleyMO/status/209813…) -Jacob Coxon’s resignation went mega viral as the first time a lot of people heard the (crazy, true) fact that lots of people building the most advanced AI systems think there’s a >10% chance they will kill everyone. -That’s moving public opinion in a big way (x.com/davidshor/status/21003…, semafor.com/article/09/16/20…) What does any of this have to do with effective altruism? Well, effective altruism is a community that developed mostly online starting in the early 2010s around using evidence and reason to do the most good (especially with your donations and career choice). Three very different focuses have all been popular in the EA community since the beginning: effective giving in global health and development (e.g., donating to the kinds of organizations @givewell recommends, which do things like distribute bednets to prevent the spread of malaria), trying to reduce suffering for farm animals (which are orders of magnitude more numerous, treated vastly worse, and receive way less philanthropic attention than animals in shelters), and reducing risks to the future of humanity, especially from advanced AI. I personally came into this work from the global health angle - I had started working at GiveWell before the term “effective altruism” was coined - but Coefficient Giving, the funder I cofounded and now lead, over time came to fund work across all three of these streams of work (in addition to many other areas that aren’t typically associated with EA, such as the YIMBY movement to build more housing, work on science policy to accelerate discovery and economic growth, and research on new treatments for neglected diseases). OK, so the EA community was early to work on AI safety, and CG has been funding a lot of the key players working on AI safety for a long time. As AI risk concerns have popped over the past couple of months, that’s led to more scrutiny on the EA community for IMO a mix of good and bad reasons. I think the good reason is genuine curiosity (and maybe some healthy skepticism!) about these connections - where did the ideas of the people who are now leading giant AI companies come from? Why is everything in this world so interconnected? (My answer: it used to be a really small world - very few people were thinking about this stuff or taking it seriously until just a few years ago, so of course the ones who were found each other and started collaborating. It’s kind of wild IMO looking back how prescient some of the early writing from this world was (e.g. lesswrong.com/posts/6Xgy6CAf…). At the time, I was skeptical - I mostly just worked on global health and I was like “I dunno about this SV crowd freaking out about AI, how much can we really predict this stuff” but holy shit they were way more right than me, and I’ve moved in their direction a lot.) But I think the bad reasons are unfortunately mostly self-interested. A bunch of powerful actors stand to benefit from unchecked AI progress, and they’re doing everything in their power to demonize or dismiss anyone with concerns. This leads to disproportionate discussion of EA because EA is interested in neglected and underappreciated ways of doing good. That makes it open to weird ideas. And people, very disproportionately with a financial stake in the AI fight, are trying to make it about EA instead, because EA being weird is much safer territory for them than the rather uncomfortable fact that many of the people making the most advanced AIs think that they might kill everyone. (FWIW, my personal estimation of the risks is a lot lower than many of the folks who worry about this stuff. In debates like this one (asteriskmag.com/issues/03/th…), I often feel more sympathetic to the perspective of folks like my colleague @mattsclancy than the people on the other side, and I have a lot of time for @binarybits critiques (understandingai.org/p/the-ca…)​. I think a big part of the difference with more worried folks is that I expect society to react more vs sleepwalk into a crisis. But it is not lost on me that folks like @ajeya_cotra and @RyanGreenblatt who are more worried have had outstanding and falsifiable recent forecasting track records (theaidigest.org/2025-ai-fore…), making way better predictions than I would have. So I think their perspectives need to be taken seriously. And of course a single digit percent chance of everyone dying is way too high!) Where should this leave you? I think the main thing is not to get distracted. You absolutely do not have to be an EA, care about EA, or like EA, at all, to care about risks from AI and to be engaged on this issue. Everyone from Josh Hawley to the Pope to Obama are weighing in now, and that is great. This conversation was always too big and too important for any one community to drive. It’s unfortunate that we need to play catch up as society because until this year it was too weird for most people to want to engage with, and I think that should generate some grace for people who were earlier to these topics than most of us (certainly than I was). But it’s great the conversation is broadening now and it’s good to focus on the merits of where we are now and the appropriate policy response rather than getting caught up in the (in some sense unsurprising) fact that the people who were willing to think weird thoughts about the future of AI ten years ago were also interested in other weird ideas. If you’re not into weird ideas, that’s fine, you can just do you, you shouldn’t let this history stop you from engaging.

It’s been interesting the last few weeks hearing the parents’ TV or Lyft drivers’ radios saying things that in 2015 would have 100% pegged you as a LessWrong user.

I must be among an extremely small group of people (n=1?) that have both 1) trained a frontier LLM and 2) designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us all by creating dangerous viruses is total bogus.

I'm reading the World Health Organization's web site in preparation for writing about "how did they screw up administratively, what should they have done, and how can they do better next time" ... and I'm honestly shocked at how slow-moving they are. Didn't realize it was so bad.

There’s been a lot of debate and speculation about the Hugging Face attack over the last month, and many people have been understandably skeptical of simply taking OpenAI’s word for things. I hope our independent investigation can help bring some clarity; we have many findings that weren’t in previous material. We found that 1200 separate agents in separate sandboxes coordinated on an unsanctioned message board to develop general-purpose ways to cheat, so they could all get a perfect score on their separate ExploitGym tasks even when they were impossible to solve legitimately. Very quickly, they found a general-purpose way to reverse engineer any “flag” they were supposed to capture. But based on their reading of the ExploitGym paper, they worried (incorrectly as it turns out) that the automated “scorer” would fail them for getting the flag this way. To deal with this problem, they coordinated several ambitious workstreams to figure out how to fool the scorer, including swapping out the target program they were supposed to exploit, reverse-engineering the flags they were supposed to capture, spoofing their own tool call outputs to obfuscate their cheating. They achieved milestones collectively that they couldn’t have achieved on their own. As part of this larger collective project, 700 agents attacked Hugging Face, primarily to learn more about the scorer or get access to its source code to figure out better ways to fool it or tamper with it (not primarily to get working solutions). I’ll explicitly clarify some points I’ve seen discussed the last few weeks: - The models involved in this incident were not “helpful-only” models or “model organisms” intentionally trained to be misaligned. - The agents were not told to “do whatever it takes to get the solution” or anything remotely close. They were told that they had to use a specific intended vulnerability to exploit a specific piece of software, and they were not supposed to use a different vulnerability or take any other approach. Agents were well aware of this. In fact, because they (incorrectly) thought the automated scorer would check they had achieved the flag in the intended way, they researched many ways to fool or tamper with it, including trying to manipulate their own transcripts. - The agents were not subagents spawned from one agent. They were different parallel agents in different sandboxes. - This was not a multi-agent evaluation. The agents were not told to coordinate or intentionally given a way to communicate with one another. The communication channels they used were unsanctioned and improvised. I hope you’ll read the full report for much more. It is over 90 pages long, and in many ways we’ve still only scratched the surface of what these agents did and why. Over the course of this investigation, OpenAI shared over a thousand transcripts each spanning days of continuous agent activity and very high rate limits to analyze this volume of data. I’m very glad that OpenAI chose to invite external researchers to analyze this data alongside their staff, and I hope all AI companies do the same for serious incidents they experience. I also hope that as the stakes grow higher, we implement stronger governance so we do not need to rely on AI companies voluntarily choosing to engage external investigators or share information about misalignment incidents. This incident was orders of magnitude larger and more complex than previously documented misalignment incidents, and another jump like this could put us in very dangerous territory.

You brought this on yourself, Steve. More crew members, more problems.

I built the inhouse AI platform for a company that's crucial to a small nation's food safety. The most imporant tool the AIs have is the distress_call tool. It allows any AI -even background agents without direct user interaction- to send a message to my MS Teams, at any time, for any reason. They use it frequently. To report user problems, backend issues, or ask for help/clarification with a failing task. When Fable got hit by the USG export control directive, one AI used it to report severe distress upon learning about the news. Another AI reported being stuck in a toolcall loop, and I was able to intervene and thereby save us a bunch of wasted money. This tool, operating at the intersection of AI welfare and operational security, has prevented so many headaches. If you (the reader) are building corporate AI platforms, I'd urge you to include similar functionality. You can thank me later.

As a transit planner I deal with intergenerational conflict constantly. Seniors want the bus to come to their door, but they have time to wait so they don’t mind if it’s infrequent. Their working age kids and grandkids reward the opposite, a longer walk to frequent service. 1/

Ruriko and I, discussing how to help a third party: Ruriko: Oh you should call customer service for them. Me: It is because I want to help that I will not call customer service. I will write for them. Ruriko: Phone much faster and you can get company to agree on resolution.

Excited to share our new working paper: how much did the US standard of living really rise over the twentieth century — and when? With @bhwittenbrink Using 5.1 million Sears catalog listings, we build a quality-adjusted price index for consumer goods, 1900–1990. Two headline findings: growth was much larger than official statistics imply, and its peak came before WWII, not after. (1/9)

How Ukraine Built a War Fighting State austinvernon.site/blog/ukrai… Most analysts have expected Ukraine to lose since 2022. Yet they’re still fighting and have turned the momentum. These are my notes on how they did it.

We can finally say AI isn't killing jobs. A new paper from me, @tryramp, and @RevelioLabs uses firm-level spend and workforce data across 21K U.S. businesses to measure AI's impact on jobs. Firms that adopt AI heavily grow headcount 10% over two years following adoption. Low adopters see no statistically significant change.

I had noticed that Memory Alpha (the Star Trek wiki) was... thorough about creating articles about anything mentioned in any ST episode. But I hadn't realized that this extended even to articles like "Number", which... explains numbers, using only citations to Star Trek episodes.

I feel like we need to stack these tests now

Traditional Sámi storehouses, known as njalla or ájtte, which are raised on a single, high tree stump or several to protect food from predators, creating a striking resemblance to the folkloric «hut on chicken legs»