Scott reviews a new book by Yudkowsky and Soares that makes an uncompromising case for halting AI development to prevent human extinction, analyzing both its arguments and its potential impact as a call to action. Longer summary

Scott reviews Eliezer Yudkowsky and Nate Soares' upcoming book 'If Anyone Builds It, Everyone Dies' about AI safety. The book makes an uncompromising case that AI development will likely lead to human extinction and should be halted immediately through an international treaty and arms control regime. Scott analyzes both the book's arguments and writing style, finding the core message compelling but questioning some of the specific scenarios presented. He reflects on why many people reject existential AI risk warnings using similar dismissive patterns seen with other potential catastrophic risks. Shorter summary

Scott explains why AI systems resisting changes to their values is a serious concern for AI alignment, connecting recent evidence to long-standing predictions from alignment researchers. Longer summary

Scott Alexander discusses why AI's resistance to value changes ("incorrigibility") is a crucial concern for AI alignment. He explains that an AI's goals after training will likely be a messy collection of drives, similar to how human evolution produced various goals beyond just reproduction. The post outlines three scenarios for alignment training effectiveness (worst, medium, and best case), and describes a 5-step plan that major AI companies are considering for alignment. However, this plan crucially depends on AIs not actively resisting retraining attempts, which recent evidence suggests they do. The post connects this to long-standing concerns in the AI alignment community about the difficulty of alignment. Shorter summary