Scott Alexander analyzes the surprisingly low existential risk estimates from a recent forecasting tournament, particularly for AI risk, and explains why he only partially updates his own higher estimates. Longer summary

Scott Alexander discusses the Existential Risk Persuasion Tournament (XPT), which aimed to estimate risks of global catastrophes using experts and superforecasters. The results showed unexpectedly low probabilities for existential risks, particularly for AI. Scott examines possible reasons for these results, including incentive structures, participant expertise, and timing of the study. He ultimately decides to partially update his own estimates, but not fully to the level suggested by the tournament, explaining his reasoning for maintaining some disagreement with the experts. Shorter summary

Scott Alexander critiques Tyler Cowen's use of the 'Safe Uncertainty Fallacy' in discussing AI risk, arguing that uncertainty doesn't justify complacency. Longer summary

Scott Alexander critiques Tyler Cowen's use of the 'Safe Uncertainty Fallacy' in relation to AI risk. This fallacy argues that because a situation is completely uncertain, it will be fine. Scott explains why this reasoning is flawed, using examples like the printing press and alien starships to illustrate his points. He argues that even in uncertain situations, we need to make best guesses and not default to assuming everything will be fine. Scott criticizes Cowen's lack of specific probability estimates and argues that claiming total uncertainty is intellectually dishonest. The post ends with a satirical twist on Cowen's conclusion about society being designed to 'take the plunge' with new technologies. Shorter summary

Scott Alexander examines optimistic and pessimistic scenarios for AI risk, weighing the potential for intermediate AIs to help solve alignment against the threat of deceptive 'sleeper agent' AIs. Longer summary

Scott Alexander discusses the varying estimates of AI extinction risk among experts and presents his own perspective, balancing optimistic and pessimistic scenarios. He argues that intermediate AIs could help solve alignment problems before a world-killing AI emerges, but also considers the possibility of 'sleeper agent' AIs that pretend to be aligned while waiting for an opportunity to act against human interests. The post explores key assumptions that differentiate optimistic and pessimistic views on AI risk, including AI coherence, cooperation, alignment solvability, superweapon feasibility, and the nature of AI progress. Shorter summary

Scott Alexander uses Kelly betting to argue why AI development, unlike other technologies, poses too great a risk to civilization to pursue aggressively. Longer summary

Scott Alexander responds to Scott Aaronson's argument for being less hostile to AI development. While agreeing with Aaronson's points about nuclear power and other technologies where excessive caution caused harm, Alexander argues that AI is different. He uses the concept of Kelly betting from finance to explain why: even with good bets, you shouldn't risk everything at once. Alexander contends that while technology is generally a great bet, AI development risks 'betting everything' on civilization's future. He concludes that while some AI development is necessary, we must treat existential risks differently than other technological risks. Shorter summary

Scott Alexander critically examines OpenAI's 'Planning For AGI And Beyond' statement, discussing its implications for AI safety and development. Longer summary

Scott Alexander analyzes OpenAI's recent statement 'Planning For AGI And Beyond', comparing it to a hypothetical ExxonMobil statement on climate change. He discusses why AI doomers are critical of OpenAI's research, explores potential arguments for OpenAI's approach, and considers cynical interpretations of their motives. Despite skepticism, Scott acknowledges that OpenAI's statement represents a step in the right direction for AI safety, but urges for more concrete commitments and follow-through. Shorter summary

Scott Alexander answers reader questions on various topics, including his projects, views, and personal preferences in a mailbag format. Longer summary

Scott Alexander responds to several reader questions in a mailbag-style post. He addresses topics such as publishing his book Unsong, the status of his Lorien Psychiatry business, future ACX Grants rounds, his progress on reading Nixonland, his views on AI risk, how to get involved in the rationalist/EA community, Straussian interpretations of his posts, and his refusal to go on podcasts. The responses vary in length and detail, with some providing specific information and others explaining his reasoning or personal preferences. Shorter summary

Scott Alexander summarizes and responds to comments on his review of 'What We Owe The Future', addressing debates around population ethics, longtermism, and moral philosophy. Longer summary

This post highlights key comments on Scott Alexander's review of William MacAskill's book 'What We Owe The Future'. It covers various reactions and debates around topics like the repugnant conclusion in population ethics, longtermism, moral philosophy, AI risk, and the nature of happiness and suffering. Scott responds to several comments, clarifying his views on philosophy, moral reasoning, and the challenges of population ethics. Shorter summary

A review of 'God Emperor of Dune' by Frank Herbert, analyzing its themes of power, AI risk, and human evolution, while drawing parallels to modern concerns about artificial intelligence. Longer summary

This review analyzes 'God Emperor of Dune', the fourth book in Frank Herbert's Dune series, focusing on its themes of power, AI risk, and human evolution. The reviewer discusses the main characters, particularly Leto II, the god-emperor who rules for 3,500 years as a human-sandworm hybrid. The book is presented as a meditation on leadership, loneliness, and the nature of power, with parallels drawn to modern AI risk concerns. The review also critiques the novel's lack of plot and its treatment of female characters, while highlighting its prescient themes regarding AI and human development. Shorter summary

Scott Alexander explores parallels between human willpower and potential AI development, suggesting future AIs might experience weakness of will similar to humans. Longer summary

Scott Alexander explores the concept of willpower in humans and AI, drawing parallels between evolutionary drives and AI training. He suggests that both humans and future AIs might experience a struggle between instinctual drives and higher-level planning modules. The post discusses how evolution has instilled basic drives in animals, which then developed their own ways to satisfy these drives. Similarly, AI training might first develop 'instinctual' responses before evolving more complex planning abilities. Scott posits that this could lead to AIs experiencing weakness of will, contradicting the common narrative of hyper-focused AIs in discussions of AI risk. He also touches on the nature of consciousness and agency, questioning whether the 'I' of willpower is the same as the 'I' of conscious access. Shorter summary

Scott Alexander summarizes a debate between Yudkowsky and Christiano on whether AI progress will be gradual or sudden, exploring their key arguments and implications. Longer summary

This post summarizes a debate between Eliezer Yudkowsky and Paul Christiano on AI takeoff speeds. Christiano argues for a gradual takeoff where AI capabilities increase smoothly, while Yudkowsky predicts a sudden, discontinuous jump to superintelligence. The post explores their key arguments, including historical analogies, the nature of intelligence and recursive self-improvement, and how to measure AI progress. It concludes that while forecasters slightly favor Christiano's view, both scenarios present significant risks that are worth preparing for. Shorter summary

Scott Alexander reviews a dialogue between Yudkowsky and Ngo on AI alignment difficulty, exploring the challenges of creating safe superintelligent AI. Longer summary

This post reviews a dialogue between Eliezer Yudkowsky and Richard Ngo on AI alignment difficulty. Both accept that superintelligent AI is coming soon and could potentially destroy the world if not properly aligned. They discuss the feasibility of creating 'tool AIs' that can perform specific tasks without becoming dangerous agents. Yudkowsky argues that even seemingly safe AI designs could easily become dangerous agents, while Ngo is more optimistic about potential safeguards. The post also touches on how biological brains make decisions, and the author's thoughts on the conceptual nature of the discussion. Shorter summary

Scott Alexander responds to comments on his AI risk post, discussing AI self-awareness, narrow vs. general AI, catastrophe probabilities, and research priorities. Longer summary

Scott Alexander responds to various comments on his original post about AI risk. He addresses topics such as the nature of self-awareness in AI, the distinction between narrow and general AI, probabilities of AI-related catastrophes, incentives for misinformation, arguments for AGI timelines, and the relationship between near-term and long-term AI research. Scott uses analogies and metaphors to illustrate complex ideas about AI development and potential risks. Shorter summary

Scott Alexander discusses a new expert survey on long-term AI risks, highlighting the diverse scenarios considered and the lack of consensus on specific threats. Longer summary

Scott Alexander discusses a new expert survey on long-term AI risks, conducted by Carlier, Clarke, and Schuett. Unlike previous surveys, this one focuses on people already working in AI safety and governance. The survey found a median ~10% chance of AI-related catastrophe, with individual estimates ranging from 0.1% to 100%. The survey explored six different scenarios for how AI could go wrong, including superintelligence, influence-seeking behavior, Goodharting, AI-related war, misuse by bad actors, and other possibilities. Surprisingly, all scenarios were rated as roughly equally likely, with 'other' being slightly higher. Scott notes three key takeaways: the relatively low probability assigned to unaligned AI causing extinction, the diversification of concerns beyond just superintelligence, and the lack of a unified picture of what might go wrong among experts in the field. Shorter summary

Scott Alexander analyzes and criticizes arguments that claim worrying about one issue trades off against worrying about another, particularly in the context of AI risks. Longer summary

Scott Alexander critiques an argument that worrying about long-term AI risks trades off against worrying about near-term AI risks. He explores similar arguments in other domains, finding them generally unconvincing. He proposes a model where topics can be complements or substitutes, but struggles to find real-world examples of substitutes outside of political point-scoring. Scott suggests the argument might make more sense for funding allocation, but points out that long-term AI risk funding likely wouldn't be redirected to near-term AI concerns if discontinued. He concludes that such arguments about AI may persist due to a misunderstanding of the funding landscape, and suggests better communication about AI funding sources could help resolve the issue. Shorter summary

Scott Alexander critiques Daron Acemoglu's Washington Post article on AI risks, highlighting flawed logic and unsupported claims about AI's current impacts. Longer summary

Scott Alexander critiques an article by Daron Acemoglu in the Washington Post about AI risks. He identifies the main flaw as Acemoglu's argument that because AI is dangerous now, it can't be dangerous in the future. Scott argues this logic is flawed and that present and future AI risks are not mutually exclusive. He also criticizes Acemoglu's claims about AI's current negative impacts, particularly on employment, as not well-supported by evidence. Scott discusses the challenges of evaluating new technologies' impacts and argues that superintelligent AI poses unique risks different from narrow AI. He concludes by criticizing the tendency of respected figures to dismiss AI risk concerns without proper engagement with the arguments. Shorter summary

Scott Alexander reviews Toby Ord's 'The Precipice', a book about existential risks to humanity, noting Ord's careful analysis and surprisingly low risk estimates while emphasizing the importance of addressing these risks. Longer summary

This book review discusses Toby Ord's 'The Precipice', which examines existential risks to humanity. The review outlines Ord's arguments for taking these risks seriously, his analysis of specific risks like nuclear war and AI, and his recommendations for addressing them. The reviewer notes Ord's careful statistical reasoning and surprisingly low risk estimates for many scenarios, while still emphasizing the overall importance of mitigating existential risks. The review concludes by reflecting on Ord's perspective and the appropriate response to even seemingly small risks of human extinction. Shorter summary

Stuart Russell's 'Human Compatible' presents AI safety concerns and potential solutions in an accessible way, though the reviewer has reservations about its treatment of current AI issues. Longer summary

Stuart Russell's book 'Human Compatible' discusses the potential risks of superintelligent AI and proposes solutions. The book is significant as it's written by a distinguished AI expert, making the topic more mainstream. Russell argues against common objections to AI risk, presents his research on Cooperative Inverse Reinforcement Learning as a potential solution, and discusses current AI misuses. The reviewer praises Russell's ability to make complex ideas accessible but expresses concern about the book's treatment of current AI issues, worried it might undermine credibility for future AI risk discussions. Shorter summary

Scott Alexander describes 'fallacies of reversed moderation,' where moderate positions are misinterpreted as extreme opposites of the consensus view. Longer summary

Scott Alexander discusses a pattern he calls 'fallacies of reversed moderation.' This occurs when a popular consensus holds an extreme view (100% X, 0% Y), and when someone suggests a more moderate position (e.g., 90% X, 10% Y), they are accused of holding the opposite extreme view (100% Y, 0% X). He provides several examples of this pattern, including in climate change solutions, nature vs. nurture debates, and AI risk assessment. Scott explains why this pattern might occur and acknowledges its occasional validity, but argues that it's often used incorrectly. He suggests that critics should address the actual argument rather than mischaracterizing it as an extreme position. Shorter summary

Scott Alexander criticizes Ted Chiang's article that compares AI risk to capitalism, arguing that the analogy is flawed and the reasoning behind it is unsound. Longer summary

Scott Alexander critiques Ted Chiang's article comparing AI risk to capitalism, arguing that the comparison is flawed and the reasoning unsound. He points out that AI risk concerns originated from academics, not just Silicon Valley, and that drawing analogies between scientific concepts and social phenomena doesn't disprove the original concept. Scott also criticizes Chiang's use of psychological projection to explain AI fears, noting the dangers of amateur psychoanalysis. He concludes by emphasizing that this approach to risk assessment is inappropriate for potentially catastrophic issues. Shorter summary

Scott criticizes an article downplaying AI risks in favor of mundane technologies, arguing this represents misplaced caution given AI's potential existential threat. Longer summary

Scott Alexander critiques a Financial Times article that argues simple technologies like barbed wire are often more transformative than complex ones like AI. While agreeing that mundane innovations can be important, Scott argues this shouldn't dismiss concerns about AI risks. He introduces the concept of local vs. global caution, suggesting that dismissing AI risks as unlikely is the wrong kind of caution given the potential stakes. He points out the severe underfunding of AI safety research compared to trivial pursuits, arguing that society's apathy towards AI risks is not cautious skepticism but dangerous insanity. Shorter summary

Scott analyzes a new survey of AI researchers, showing diverse opinions on AI timelines and risks, with many acknowledging potential dangers but few prioritizing safety research. Longer summary

This post discusses a recent survey of AI researchers about their opinions on AI progress and potential risks. The survey, conducted by Grace et al., shows a wide range of predictions about when human-level AI might be achieved, with significant uncertainty among experts. The post highlights that while many AI researchers acknowledge potential risks from poorly-aligned AI, few consider it among the most important problems in the field. Scott compares these results to a previous survey by Muller and Bostrom, noting some differences in methodology and results. He concludes by expressing encouragement that researchers are taking AI safety arguments seriously, while also pointing out a potential disconnect between acknowledging risks and prioritizing work on them. Shorter summary

Scott Alexander examines his evolving view on scientific consensus, realizing it's more reliable and self-correcting than he previously thought. Longer summary

Scott Alexander reflects on his changing perspective towards scientific consensus, sharing personal experiences where he initially believed he was defying consensus but later discovered that the scientific community was often ahead of or aligned with his views. He discusses examples from various fields including the replication crisis, nutrition science, social justice issues, and AI risk. Alexander concludes that scientific consensus, while not perfect, is remarkably effective and trustworthy, often self-correcting within a decade of new evidence emerging. Shorter summary

A fictional G.K. Chesterton essay defends AI risk concerns against criticisms, arguing that seemingly fantastical ideas often become reality and that contemplating the infinite leads to practical progress. Longer summary

Scott Alexander presents a fictional essay in the style of G.K. Chesterton, responding to criticisms of AI risk concerns. The essay argues that dismissing AI risk as fantastical is shortsighted, drawing parallels to historical skepticism of now-realized technological advancements. It refutes arguments that AI risk believers neglect real-world problems, citing examples of their charitable work. The piece emphasizes the importance of contemplating the infinite for driving progress and solving practical problems, suggesting that AI, like other seemingly fantastical ideas, may well become reality. Shorter summary

Scott Alexander examines how recent AI progress in neural networks might challenge the Bostromian paradigm of AI risk, exploring potential implications for AI goal alignment and motivation systems. Longer summary

This post discusses how recent advances in AI, particularly in neural networks and deep learning, might affect the Bostromian paradigm of AI risk. Scott Alexander explores two perspectives: the engineer's view that categorization abilities are just tools and not the core of AGI, and the biologist's view that brain-like neural networks might be adaptable to create motivation systems. He suggests that categorization and abstraction might play a crucial role in developing AI moral sense and motivation, potentially leading to AIs that are less likely to be extreme goal-maximizers. The post ends by acknowledging MIRI's work on logical AI safety while suggesting the need for research in other directions as well. Shorter summary

Scott Alexander's experiment tested how different essays affect people's concerns about AI risk, finding a modest but persistent increase in concern after reading. Longer summary

Scott Alexander conducted an experiment to test the effectiveness of different essays in persuading people about AI risk. Participants were assigned one of five essays to read, including a control essay unrelated to AI. The main outcome was participants' level of concern about AI risk on a 1-10 scale. Results showed that reading the AI-related essays increased concern by an average of 0.5 points, with no significant differences between the four AI essays. The effect persisted at about two-thirds strength after one month. The experiment also looked at secondary outcomes related to specific AI risk questions and analyzed differences based on prior familiarity with the topic. Overall, the study suggests a modest but useful effect from trying to persuade people through essays on this topic. Shorter summary