Research Replication Revolution

By: Sam Ernst, Meg Wang, and Erin Dunn

Ignited by reports of “unacceptable low levels” of statistical power and astonishingly high rates of false positives (>50%) in psychology papers (Szucs & Ioannidis, 2017), scientists in psychology and many other disciplines have started a “replication revolution" designed to increase confidence in scientific findings.  The goal is to encourage researchers to replicate their findings - meaning do the study again (with different research participants and often with different research conditions) to determine if the basic findings of the original study hold-up. Although many agree that replication is good for science - see for example recent work in Slate and the Atlantic - some recently published articles, including one about replication in psychology by Sabeti, a Boston-based researcher, offer a word of caution towards this so-called “revolution”. What do we think?  

1. Replication is the basis of good science.

When asked about the replication movement, Daniel Engber, a science writer for the Slate, the New York Times, and Wired, remarked: “It’s grim to watch a theory get dismantled, even when that theory happens to be wrong. But that’s how science works.” We couldn’t agree more. This is why we often try to replicate our own findings.

2. Replication makes us scientists work together more, which leads to better science.

Brian Nosek, a psychologist from the University of Virginia, persuaded over 270 of his peers to join his initiative -- The Reproducibility Project -- to repeat 100 published psychology experiments to see if they could get the same results a second time around. Although this project exposed a gaping hole in the reproducibility of psychological research and drew even more questions surrounding the meaning of these findings, Nosek remarked that: “It has been a fantastic experience, all this common energy around a very specific goal...the collaborators all contributed their time to the project knowing that they wouldn’t get any credit for being 253rd author.”

3.  Replication may ultimately help increase public trust in science.  

Scientific findings can be met with skepticism.  Just think of all those news headlines suggesting “eating chocolate reduces your risk of [insert your favorite disease]”. These sorts of headlines - though attention-grabbing - oftentimes do a disservice to the field because they lead the public to think that a scientific finding is 100% true, even when it has been poorly tested, and especially when a second study comes out the following week to say that “eating chocolate is bad for your health”.  Public skepticism around the believability of scientific findings can have many dangerous and long-term consequences. Efforts to replicate study findings – and publicize only those findings that have been highly scrutinized – could go a long way to increase public trust in science.

4. But replication is really hard to do.  

Even when the interest is there, it can be a challenge to replicate prior study findings.  For example, in our studies of gene-environment interaction, it is often difficult to find other studies with the exact same measures of stress exposure.  And when you introduce a different measure - and observe a different finding - it’s hard to know whether the replication failed or the finding just doesn’t generalize to another measure of stress, or population, or setting. However, the challenge of replication should not be viewed as a hopeless situation.  In fact, researchers like Shepherd are already proposing pragmatic approaches to make research that is difficult to share or replicate quasi-reproducible.

All that said, we think the replication movement is good for science.  So viva la revolution!