A replication of a 2023 neuroscience study on brain entrainment and learning found that the original effect likely doesn't exist, revealing how the study obscured key issues through statistical averaging and highlighting how AI tools are democratizing scientific scrutiny. Longer summary

Sasha Putilin received an ACX grant to replicate a 2023 study claiming that flickering lights synchronized to individual brain rhythms could boost learning speed threefold. The $32,000 replication with 12 participants (versus 80 in the original) found no such effect. Upon examining the original data more closely, Putilin discovered the apparent effect was driven entirely by a few participants with large negative learning rates (getting worse over time) who happened to all be in one experimental group. The original study had obscured this pattern by presenting averaged data rather than individual results. Putilin argues this exemplifies 'cargo-cult statistics' where researchers mechanically apply statistical rituals without critically examining underlying data. The post concludes by suggesting AI tools are democratizing meta-science, enabling anyone to audit published research that previously required expert-level effort. Shorter summary