aurora network logo

Reproducibility Crisis Overestimated

VU International News and Reviews No. 110 April 9 2018

April 2018

Two researchers from Stanford and one from the Leiden CWTS have conducted a major meta-analysis of Web of Science articles to search for evidence of the “reproducibility crisis caused by many biases”. Their report on “Meta-assessment of bias in science” in the Proceedings of the National Academy of Science shows that the actual situation may be less worrisome than people might believe. They looked for evidence of a great number of possible causes for distortion of research results – reporting stronger positive effects than warranted because allegedly:

  • The small size of the study
  • Studies with negative effects don’t get published
  • Early studies in a specific topic give more extreme results
  • Studies with positive results get more citations
  • Publication pressure – especially in the US – leads to exaggerated results
  • Young – particularly male – researchers are ambitious and want to shine
  • In some fields, there is a lack of peer control

All in all, they found that these biases, where they exist, are relatively small. They found some inclination for overestimated effects in small, early and high-cited articles. But their overall conclusion that these biases are much smaller than the public discussion on the reproducibility crisis would make it seem. Rather than investing heavily in this assumed crisis, they suggest that the feasibility and costs of interventions to attenuate biases in the literature might need to be discussed on a discipline-specific and topic-specific basis.

Back to newsletters