Just for fun, let's suppose it's a few hundred years ago and you've been experimenting in the kitchen and made a scientific breakthrough: Water boils at 212 degrees Fahrenheit at sea level. Before your research, it was a guessing game. Now, water's boiling point is known. You're a genius.
Also, just for fun, let's suppose your neighbor repeats your experiment and comes up with a different temperature. Now everybody is talking again, but this time they've got questions.
First, what's the true boiling point? Second, was the first breakthrough bogus, or the follow-up? What's accurate?
This replication problem is similar to what's going on in the social sciences today, only it's sometimes called a "replication crisis." Other names are replicability crisis or reproducibility crisis. Either way, the concern is the same: In some cases, repeats of earlier headline-grabbing research printed in textbooks across the globe don't produce the same results. Replication is a core principle of the scientific method, where two things are necessary:
Validity: The study measures what it purports to measure. And reliability: The study can be reproduced. If one of those two things is missing, something's wrong. A project called Many Labs 2 focuses on one social science field, psychology.
Almost 200 psychologists have tried to repeat peer-reviewed, published experiments. The results: Out of 28 experiments, 14 were replicated. Replication of some long-standing, high-profile experiments has failed. Included: studies finding that subliminal exposures influence behavior; that people have a finite supply of willpower that can be worn down; and that the expression on our face influences our mood.
So why is this happening? It's science and therefore complicated. There are a lot of possible variables: The replication researchers might have made a mistake.
The original research might have fallen victim to a false positive. The subjects of experiments — meaning human beings — can change between the time of the original experiment and the replication effort. The evolution, presence and influence of technology, for example, could make a difference.
Another possibility: Pressure at universities for researchers to conduct a lot of experiments and write a lot of papers. That's how you get tenure. This has the potential to result in research based on small samples, fast work and hasty conclusions.
There is one other possibility: Fraud. In other words, research that is manipulated for career advancement or to satisfy the goals of sponsors. There are efforts underway to end the crisis. First, quite simply, do more replication studies. But those aren't always career-boosters for professors trying to get tenure.
Second: Preregister study designs with journals where the results will be published.
Third: Some journals are demanding larger subject pools.
Fourth: Some journals are requiring that researchers share all underlying data.
By the way, back tothat fun little story about the boiling water — if for some reason your neighbor's house is a lot higher than yours, then water in his kitchen would boil at a lower temperature. Both are true.
Like we said, science is complicated. That's why replication is important.
