Scott Alexander examines two cases of multiple hypothesis testing problems in medical and social science research, highlighting the complexities in interpreting results.
Longer summary
Scott Alexander discusses two cases of multiple hypothesis testing problems. The first involves a Vitamin D study for COVID-19 where a significant difference in blood pressure between groups complicates the interpretation of results. The second case relates to Scott's own study on ambidexterity and authoritarianism, where he questions the applicability of traditional multiple hypothesis testing corrections. He explores the complexities of interpreting multiple tests of the same hypothesis and considers Bayesian approaches, ultimately acknowledging the limits of his statistical knowledge on this seemingly simple question.
Shorter summary