Science’s Inference Problem: When Data Doesn’t Mean What We Think It Does

James Ryerson in the New York Times:

Over the past few years, many scientific researchers, especially those working in psychology and biomedicine, have become concerned about the reproducibility of results in their field. Again and again, findings deemed “statistically significant” and published in reputable journals have not held up when the experiments were conducted anew. Critics have pointed to many possible causes, including the unconscious manipulation of data, a reluctance to publish negative results and a standard of statistical significance that is too easy to meet.

In their book TEN GREAT IDEAS ABOUT CHANCE (Princeton University, $27.95), a historical and philosophical tour of major insights in the development of probability theory, the mathematician Persi Diaconis and the philosopher Brian Skyrms emphasize another possible cause of the so-called replication crisis: the tendency, even among “working scientists,” to equate probability with frequency. Frequency is a measure of how often a certain event occurs; it concerns facts about the empirical world. Probability is a measure of rational degree of belief; it concerns how strongly we should expect a certain event to occur. Linking frequency and probability is hardly an error. (Indeed, the notion that in large enough numbers frequencies can approximate probabilities is Diaconis and Skyrms’s fourth “great idea” about chance.) But failing to distinguish the two concepts when testing hypotheses, they warn, “can have pernicious effects.”

More here.