Scott examines the debate over open-weights AI, explaining why he remains neutral despite risks of criminal misuse, arguing that waiting for inevitable incidents is more strategic than preemptively burning political capital fighting the strong pro-open-weights coalition. Longer summary

Scott discusses the debate over open-weights AI (where AI model weights are publicly available, similar to open-source software). He explains that while open weights AI offers freedom from corporate control, it also enables criminal misuse like hacking and bioterrorism. Despite this being a reasonable concern for AI safety advocates, Scott argues most remain neutral because waiting for the first incidents to occur (rather than preemptively fighting a political battle) is the more strategic approach. He distinguishes between existential risks from superintelligence (which require preemptive action) and criminal misuse risks (which will trigger government response after initial incidents). Scott concludes that the open-weights community deserves a chance to prove their 'good guy with an AI' defense theory can work, even though he's skeptical, because fighting them preemptively would waste political capital on a likely-losing battle. Shorter summary

Scott Alexander discusses a new expert survey on long-term AI risks, highlighting the diverse scenarios considered and the lack of consensus on specific threats. Longer summary

Scott Alexander discusses a new expert survey on long-term AI risks, conducted by Carlier, Clarke, and Schuett. Unlike previous surveys, this one focuses on people already working in AI safety and governance. The survey found a median ~10% chance of AI-related catastrophe, with individual estimates ranging from 0.1% to 100%. The survey explored six different scenarios for how AI could go wrong, including superintelligence, influence-seeking behavior, Goodharting, AI-related war, misuse by bad actors, and other possibilities. Surprisingly, all scenarios were rated as roughly equally likely, with 'other' being slightly higher. Scott notes three key takeaways: the relatively low probability assigned to unaligned AI causing extinction, the diversification of concerns beyond just superintelligence, and the lack of a unified picture of what might go wrong among experts in the field. Shorter summary