Auditing Data for Studies

Interesting kind of reliability analysis study.

'Auditing' tool can improve reliability of studies that explore relationships between things

By EKATERINA PESHEVA March 2, 2022 Research

Harvard Medical School News & Research, Ekaterina Pesheva, March 2, 2022

Harvard Medical School (HMS) scientists created the vibration of effects (VoE) auditing tool to improve the reliability of studies that explore the relationships between things. "At its most basic, the vibration of effects model analyzes how the modeling choices a researcher makes can influence what they will discover," explained former HMS researcher Braden Tierney. The tool applies brute-force computation to test the reliability of research findings, and researchers can use it to vet their own results before submission for publication. The tool was used to analyze connections between various gut microbes and six diseases in 15 published studies. High-scoring studies were found to be less reliable because their VoE results exhibited significant variation when run through multiple testing models, while low-scoring studies were found to be more reliable because they identified associations that remained consistent even when processed by different models. ...