Scott Alexander examines two cases of multiple hypothesis testing problems in medical and social science research, highlighting the complexities in interpreting results. Longer summary

Scott Alexander discusses two cases of multiple hypothesis testing problems. The first involves a Vitamin D study for COVID-19 where a significant difference in blood pressure between groups complicates the interpretation of results. The second case relates to Scott's own study on ambidexterity and authoritarianism, where he questions the applicability of traditional multiple hypothesis testing corrections. He explores the complexities of interpreting multiple tests of the same hypothesis and considers Bayesian approaches, ultimately acknowledging the limits of his statistical knowledge on this seemingly simple question. Shorter summary

Scott Alexander critiques a meta-analysis supporting psychic phenomena to illustrate flaws in scientific methodology and meta-analysis. Longer summary

Scott Alexander examines a meta-analysis by Daryl Bem that claims to provide strong evidence for psychic phenomena (psi). While Bem's analysis follows many best practices for scientific rigor, Alexander argues it likely suffers from experimenter effects and other biases that can produce false positive results. He uses this to illustrate broader issues with the scientific method and meta-analysis, concluding that even seemingly rigorous studies and meta-analyses can produce incorrect conclusions. This challenges the idea that scientific consensus and meta-analysis are the highest forms of evidence. Shorter summary