Scott Alexander examines how recent AI progress in neural networks might challenge the Bostromian paradigm of AI risk, exploring potential implications for AI goal alignment and motivation systems. Longer summary

This post discusses how recent advances in AI, particularly in neural networks and deep learning, might affect the Bostromian paradigm of AI risk. Scott Alexander explores two perspectives: the engineer's view that categorization abilities are just tools and not the core of AGI, and the biologist's view that brain-like neural networks might be adaptable to create motivation systems. He suggests that categorization and abstraction might play a crucial role in developing AI moral sense and motivation, potentially leading to AIs that are less likely to be extreme goal-maximizers. The post ends by acknowledging MIRI's work on logical AI safety while suggesting the need for research in other directions as well. Shorter summary