Making Sense of Contradictory Science Results

How do we make make sense of contradictory results?

How to Make Sense of Contradictory Science Papers  in Nautil.us

Published research is less about conclusions than science at play.

BY HAIXIN DANG & LIAM KOFI BRIGHT

The science you can come across today can often appear to be full of contradictory claims. One study tells you red wine is good for your heart; another tells you it is not. Over the past year, COVID-19 research has offered conflicting reports about the overall effectiveness of wearing a mask. As scientists debate what policy best suits the current moment, they will be drawing on hundreds of studies; some that say masks are effective1 and some that say masks alone are not enough.

Naturally, given its outsized influence on society—especially during a pandemic—people tend to regard published science highly. This means that many of us expect scientists to be prudent in reporting their results. These ought to be true and justified by evidence, right? And surely, at a bare minimum, the researchers themselves ought to believe in what they are publishing, yes? Maybe not. The bar for publishing might, counter-intuitively, be lower than one might expect. “Scientific conclusions,” as we titled our recent paper,3 “need not be accurate, justified, or believed by their authors.”

Why might it be worth worrying about how and when scientists decide to share their work?

We’re not saying scientists generally lie about their published results (this has nothing to do with misconduct). Rather, we argue that scientific papers fulfill a useful social role by doing more than merely reporting on true discoveries. It’s enough for them to draw attention to an idea that is worth pursuing further—and an idea need not be true, well-justified given all our evidence, nor even believed by the scientist in order to pass that test. The peer-review process is, in fact, designed, not to detect fraud or data manipulation, but to select for what is noteworthy.4 What is considered unexpected and thought-provoking will not always track our all-things-considered judgments of what is true, but local community standards of best scientific practice having to do with how to go about data gathering and statistical testing.

For scientists to collectively inquire effectively, they need to communicate interesting ideas to each other that are worth pursuing. Consider Avi Loeb, a theoretical astrophysicist who proposed the provocative hypothesis, not without some supporting data, that ‘Oumuamua wasn’t a comet but an alien light-sail. He, presumably, knew that more data would need to be gathered, and a more thorough study would need to be conducted, before the hypothesis could be justifiably believed.

Nonetheless, perhaps it was appropriate for Loeb to publish his data and his hypothesis. He himself might even be agnostic toward the truth of the hypothesis. He likely knew that most of his colleagues would dispute his interpretation of the data, and with good cause. In spite of all this, it was still valuable for him to publicly communicate the possibility of a new hypothesis, because it can—and maybe actually did—spur more research into, and garner attention for, astronomy. Publishing those findings was not about communicating the truth but about saying that there is something exciting and interesting that requires further inquiry.  ... "