Evaluating Replicability: Considerations for Analyses and Implications for Design
Presenter
March 7, 2022
Keywords:
- replication
- statistics
- meta-analysis
- Study design
MSC:
- 62-00
Abstract
As high-profile empirical research has questioned the replicability of scientific findings, it has become clear that there is no standard approach to designing and analyzing studies to evaluate replication. Ambiguity regarding key estimands for “replication” and the purpose of replication research has shaped statistical treatments of the topic and sparked debate in several fields. This talk sheds light on this ambiguity by identifying different possible statistical definitions of “replication” that could be studied. It then highlights relevant analysis methods and derives their statistical properties. Finally, it connects these properties to key implications for the design of primary studies, as well as subsequent replication attempts.