Scott Alexander draws parallels between OpenAI's GPT-2 language model and human dreaming, exploring their similarities in process and output quality. Longer summary

Scott Alexander compares OpenAI's GPT-2 language model to human dreaming, noting similarities in their processes and outputs. He explains how GPT-2 works by predicting next words in a sequence, much like the human brain predicts sensory input. The post explores why both GPT-2 and dreams produce narratives that are coherent in broad strokes but often nonsensical in details. Scott discusses theories from neuroscience and machine learning to explain this phenomenon, including ideas about model complexity reduction during sleep and comparisons to AI algorithms like the wake-sleep algorithm. He concludes by suggesting that dream-like outputs might simply be what imperfect prediction machines produce, noting that current AI capabilities might be comparable to a human brain operating at very low capacity. Shorter summary