A long-held belief in the scientific world is if the results of an experiment can be reproduced, then the results of the experiment are true. Berna Devezer, a University of Idaho professor of marketing, is on a team researching whether this assumption holds true.
Devezer, Bert Baumgaertner, Erkan Buzbas and Gustavo Nardin have formed an interdisciplinary team to investigate why scientists fail to reproduce others’ experiments.
The preliminary results show little connection between the reproducibility of an experiment and whether or not the results of that experiment are true, Devezer said. The kind of research Devezer and her team are investigating is called “metaresearch,” or “metascience.” This relatively new field investigates the methods used in scientific investigation and how they can be improved in the future.
“We want to understand whether irreproducibility and making a discovery or not making a discovery are related,” Devezer said. “We are trying to understand the relationship between these things, and we’ve found that the relationship is not straightforward. It’s quite complicated.”
Berna Devezer | Argonaut
The team is using a type of statistical model that deals with abstract mathematical concepts, called a stochastic model, to model the scientific process. They use well-known statistical results from statistical theory as data for the model, Devezer said.
“We extend what we find from statistical theory using agent-based models,” Devezer said. “These are computation models where you assimilate the interaction between individuals in a given community. Let’s say you build a community of virtual scientists; we look at how these scientists do their science.”
The team uses indicators such as the speed of discovery, reproducibility of experiments and how long the scientific community agrees upon a finding as true to find whether or not reproducibility of experiments is linked to how truthful their results are. The speed of discovery is how many experiments are required before the scientific community discovers the true answer to a particular question.
“We imagine that there is a universe of different models about reality — one of these is actually the true model from which we collect data,” Devezer said. “Scientists are making searches by running experiments in this universe and, depending on how big that universe is, the discovery is going to be that much slower. The speed of discovery is how quickly the true model that exists in this universe is going to become the scientific consensus.”
According to Devezer, the results the team has uncovered have not been what the scientific community expected. Although the connections between the reproducibility of experiments and how true their results are is incredibly complicated, the team has found little to no correlation between the two, Devezer said.
“Some people are really interested because it’s so novel and some are really resistant and feel that ‘no, this has to be wrong,’” Devezer said. “We would appreciate more support and interest in this research because we believe it is important.”
The team’s diverse backgrounds have aided them in researching their question in ways that would not have been possible had they been working alone. Devezer said their team is “how interdisciplinary research should work” and their model of diversity and interdisciplinary work should be “the future of science.”
The team’s paper was originally submitted to the Cornell University Library on March 27, 2018. The paper has since been edited and was republished on April 4.