Scott argues that dismissing AI as 'just a next-token predictor' is like dismissing humans as 'just reproduction machines' - both confuse the optimization process that shaped an entity with how that entity actually thinks. Longer summary

Scott argues that dismissing AI as 'just a next-token predictor' confuses levels of optimization. He draws an analogy to humans: just as humans were shaped by evolution optimizing for reproduction but don't think about sex when doing math, AIs were shaped by next-token prediction but don't simply predict tokens when thinking. Scott explains that human brains use predictive coding (predicting next sense-data) to build world-models, while AIs use next-token prediction to build their own world-models. Both processes create complex internal representations - like helical manifolds in 6D space for AIs, or toroidal attractors in human hippocampi - that operate far above the level of simple prediction. The post concludes that both humans and AIs perform 'real thought' using structures created by their respective optimization processes. Shorter summary