Which scientific results can we trust? Evidence from replications and prediction markets.
Why are there so many false results in the published scientific literature? And what is the actual share of results that do not replicate in different literatures? I will discuss these questions, with focus on culprits such as low statistical power, the many ‘researcher degrees of freedom’ and misleading p-values. I will go through some recent large replication projects in mainly psychology and economics, where my coauthors and I have redone experiments published in high impact journals with new and larger samples to see whether the main result replicates. We define replication success as a replication effect in the same direction as the original study with a p-value less than 0.05 in a two-sided test, but we also discuss other measures like relative effect sizes. In these projects we have also set up prediction markets, where researchers have been given monetary endowments to bet on the binary outcome of the replications. We interpret prediction market prices as the probability the market assigns the study successfully replicating, and use these prices to study the probability that the tested hypotheses are true at different stages of the testing process. Most of these previous projects have focused on direct replications, where we have tried to keep the replication as close as possible to the original study. I will also discuss recent work on conceptual replications where we added a forecasting survey to see whether researchers can predict results for the same hypothesis that is tested in different ways. While my work is mainly experimental, there are reasons to below that the problems are worse in non-experimental work, which is something I will discuss further. I will also discuss ways to increase the reliability of scientific results, with focus on pre-analysis plans.
Anna Dreber Almenberg is the Johan Björkman professor of economics at the Stockholm School of Economics and she is also a Wallenberg Scholar. Her interdisciplinary research is mainly in meta-science and focuses the reproducibility of scientific results. Her research has been published in top science journals including Nature, Science, Proceedings of the National Academy of Sciences, as well as top journals in economics and psychology such as American Economic Review, Psychological Science and Management Science, and has been cited more than 10,000 times.