Become a Psychologist

Yesterday, Science published a blockbuster article about the state of, well, science. Since 2011, a group called the Open Science Collaboration, headed by the psychologist Brian Nosek, has been working to replicate 100 studies previously published in leading psychology journals — that is, to conduct these experiments again to see whether the same results would pop up. (The researchers set out to replicate even more, but for various reasons the final number was culled to 100.)

What they found was rather alarming: In more than half of the studies the researchers attempted to redo, different results popped up, and among the ones that did replicate similar results, the findings were significantly less impressive than what was published the first time around. Some findings that had garnered widespread attention and sexy media headlines, including one on how women’s menstrual cycles affect their mate preferences and another on how a belief in free will influences people’s ethical decisions, failed to replicate (we’ll explain a bit more about these in a bit).

To understand why all this matters — and what it says about the exciting new science findings flashing into your news feeds seemingly every day — requires a bit of background. Here’s an explainer.

I’ve been hearing a lot about this “replication crisis” lately. Can you explain what the deal is in a non-nerdy way?

Basically, there’s an increasing awareness that many of the scientific findings we’ve accepted as “true” may be less sturdy than everyone thinks. It turns out that there are many ways that researchers, claiming to have found a given effect (“effect” just means that X does something to Y — ibuprofen reduces headache pain, for example), may have overstated or fabricated the effect they’re reporting.

Overstated effects? Fabricated ones? Sounds like there’s massive fraud afoot!

No one really thinks that. While instances of true nefariousness such as all-out data fabrication do occur, there’s a general consensus that it’s pretty rare (for one thing, most people aren’t that unethical; for another, if you’re caught, that’s it for your scientific career). What’s more common is that subtle forms of bias and pernicious incentives creep into the scientific process.