sometimes people say "sure the Effective Altruists are weird, but that doesn't mean they're wrong about AI risks" But the point is that this community has values & ethics totally anathema to the American public, and their AI policy proposals are shaped by those values & ethics.

If your feed is anything like mine, you might have been hearing more about effective altruism this week. I think the key underlying reason for the surge is that concern about the risks from advanced AI have been getting more airtime recently, and EA was early to those concerns. What’s driving the surge in attention on AI risks? -OpenAI's Hugging Face attack (openai.com/index/hugging-fac…) -METR/Redwood’s investigation of the hack came out, and (correctly IMO) freaked a lot of people out. (e.g. nitter.net/binarybits/status/2092…, nitter.net/SenJohnKennedy/status/…, nitter.net/BarackObama/status/210…, nitter.net/HawleyMO/status/209813…) -Jacob Coxon’s resignation went mega viral as the first time a lot of people heard the (crazy, true) fact that lots of people building the most advanced AI systems think there’s a >10% chance they will kill everyone. -That’s moving public opinion in a big way (nitter.net/davidshor/status/21003…, semafor.com/article/09/16/20…) What does any of this have to do with effective altruism? Well, effective altruism is a community that developed mostly online starting in the early 2010s around using evidence and reason to do the most good (especially with your donations and career choice). Three very different focuses have all been popular in the EA community since the beginning: effective giving in global health and development (e.g., donating to the kinds of organizations @givewell recommends, which do things like distribute bednets to prevent the spread of malaria), trying to reduce suffering for farm animals (which are orders of magnitude more numerous, treated vastly worse, and receive way less philanthropic attention than animals in shelters), and reducing risks to the future of humanity, especially from advanced AI. I personally came into this work from the global health angle - I had started working at GiveWell before the term “effective altruism” was coined - but Coefficient Giving, the funder I cofounded and now lead, over time came to fund work across all three of these streams of work (in addition to many other areas that aren’t typically associated with EA, such as the YIMBY movement to build more housing, work on science policy to accelerate discovery and economic growth, and research on new treatments for neglected diseases). OK, so the EA community was early to work on AI safety, and CG has been funding a lot of the key players working on AI safety for a long time. As AI risk concerns have popped over the past couple of months, that’s led to more scrutiny on the EA community for IMO a mix of good and bad reasons. I think the good reason is genuine curiosity (and maybe some healthy skepticism!) about these connections - where did the ideas of the people who are now leading giant AI companies come from? Why is everything in this world so interconnected? (My answer: it used to be a really small world - very few people were thinking about this stuff or taking it seriously until just a few years ago, so of course the ones who were found each other and started collaborating. It’s kind of wild IMO looking back how prescient some of the early writing from this world was (e.g. lesswrong.com/posts/6Xgy6CAf…). At the time, I was skeptical - I mostly just worked on global health and I was like “I dunno about this SV crowd freaking out about AI, how much can we really predict this stuff” but holy shit they were way more right than me, and I’ve moved in their direction a lot.) But I think the bad reasons are unfortunately mostly self-interested. A bunch of powerful actors stand to benefit from unchecked AI progress, and they’re doing everything in their power to demonize or dismiss anyone with concerns. This leads to disproportionate discussion of EA because EA is interested in neglected and underappreciated ways of doing good. That makes it open to weird ideas. And people, very disproportionately with a financial stake in the AI fight, are trying to make it about EA instead, because EA being weird is much safer territory for them than the rather uncomfortable fact that many of the people making the most advanced AIs think that they might kill everyone. (FWIW, my personal estimation of the risks is a lot lower than many of the folks who worry about this stuff. In debates like this one (asteriskmag.com/issues/03/th…), I often feel more sympathetic to the perspective of folks like my colleague @mattsclancy than the people on the other side, and I have a lot of time for @binarybits critiques (understandingai.org/p/the-ca…)​. I think a big part of the difference with more worried folks is that I expect society to react more vs sleepwalk into a crisis. But it is not lost on me that folks like @ajeya_cotra and @RyanGreenblatt who are more worried have had outstanding and falsifiable recent forecasting track records (theaidigest.org/2025-ai-fore…), making way better predictions than I would have. So I think their perspectives need to be taken seriously. And of course a single digit percent chance of everyone dying is way too high!) Where should this leave you? I think the main thing is not to get distracted. You absolutely do not have to be an EA, care about EA, or like EA, at all, to care about risks from AI and to be engaged on this issue. Everyone from Josh Hawley to the Pope to Obama are weighing in now, and that is great. This conversation was always too big and too important for any one community to drive. It’s unfortunate that we need to play catch up as society because until this year it was too weird for most people to want to engage with, and I think that should generate some grace for people who were earlier to these topics than most of us (certainly than I was). But it’s great the conversation is broadening now and it’s good to focus on the merits of where we are now and the appropriate policy response rather than getting caught up in the (in some sense unsurprising) fact that the people who were willing to think weird thoughts about the future of AI ten years ago were also interested in other weird ideas. If you’re not into weird ideas, that’s fine, you can just do you, you shouldn’t let this history stop you from engaging.

Some bittersweet news to announce today: I recently left @IFP to join @coeff_giving as Managing Director of Public Policy, where I'm building a new US AI policy team, overseeing the Abundance and Growth Fund alongside @mattsclancy, and managing CG's government affairs work. Building IFP has been the defining professional project of my life, and this was one of the hardest decisions I've ever made. In just four and a half years, our team became, pound-for-pound, the most effective think tank in DC. I feel insanely proud of the work we’ve done and the incredible team we’ve assembled. I sometimes joke that when @alecstapp and I launched IFP, we felt like two kids in a trench coat pretending to be a think tank. And now we're a proper institution! But it feels possible to step back now because they've hit escape velocity. The talent density at IFP is bonkers. And I have complete confidence they'll keep racking up counterfactual policy wins with Alec at the helm and our superstar directors. I'm staying on the IFP board and staying in DC. In some sense the new role is a continuation of the old one. All of IFP's policy issues are reflected in CG's portfolio, but now I'm working at a new layer of the stack. So why leave? Because AI is hitting Washington like a tsunami, and DC is still radically underprepared. I hold a lot of uncertainty about timelines, but it seems very plausible that the next 2–10 years will bring the fastest technological upheaval we've ever had to navigate. The new team I’m leading is a bet on how to prepare: proactively scanning the horizon, identifying gaps in the policy ecosystem, headhunting founders, and launching new organizations, while strengthening the democratic institutions that will have to steer through the transition to powerful AI systems. I've written an essay laying out the larger vision here: calebwatney.substack.com/p/a… There is no master plan or silver bullet here. I suspect getting AI "right" is going to feel more like a chaotic, iterative process of institutions trying to make better decisions over time as the facts change underneath them. As John von Neumann wrote in 1955 about mastering an earlier technological revolution: “What safeguard remains? Apparently only day-to-day — or perhaps year-to-year — opportunistic measures, a long sequence of small, correct decisions.” Each of the small, correct decisions ahead will look small only in the sweep of the full historical picture. Up close, every one of them will require heroic levels of effort and coordination. Coefficient Giving is scaling rapidly to meet the moment, part of what Nan Ransohoff has called the “third wave of American philanthropy”, potentially large enough to fund thousands of new projects and organizations. The binding constraint is unlikely to be money. It will be people: grantmakers and policy entrepreneurs and others with the judgment to make a long sequence of small, correct decisions, and the ambition to build the institutions we wish we had. I'm hiring a team of exactly those people, starting with generalist grant makers and a chief of staff. If you share this vision, please apply! And if you are building something that we’ll need in the years ahead, reach out. jobs.ashbyhq.com/coefficient…

Just to update this chart: 1) AI salience has increased dramatically in the past week - increasing as much in the last week as the previous year combined 2) 80% of voters think it's either very or somewhat likely that AI will cause widespread job loss in the five to ten years 3) 64% of voters think it's either very or somewhat likely that AI could pose a threat to humanity's survival 4) Large bipartisan majorities back immediate government action on AI even when primed about risk from China

We’ve run the most comprehensive series of studies on expert AI forecasts over the last four years. Today, we’re sharing an interim update on our findings about the accuracy of these forecasts. Our major findings are: 1. Experts, including top economists, computer scientists, and biologists, have dramatically underestimated AI capabilities progress each year. Superforecasters have underestimated progress to an even greater extent. 2. Experts have a more mixed forecasting track record on AI diffusion-related measures, with some major underestimates (forecasting AI revenue) and other forecasts on track to be accurate (the share of electricity used for AI). 3. Some notable cases of overestimating AI progress: how much mid-2025 AI models could help amateurs do biorisk-relevant laboratory tasks; the speed of rollout of self-driving cars. 4. It is too soon to say how forecasters have performed at predicting macro-scale impact on outcomes such as GDP growth, major AI harms, and averted deaths from disease. More 🧵

A majority of Democrats want to pack the Supreme Court and abolish the filibuster, but getting rid of the Senate altogether is highly disfavored (however, 34% of Dems under 45 support it).

Interesting back and forth. Ezra describes the HF incident, Jensen says "well they shouldn't release the product." Ezra says "this product wasn't released," and Jensen's response is that if they say they can't contain their experiments then "we have to shut the labs down"