This is why you shouldn’t believe that exciting new medical study

Julia Belluz in Vox:

In 2003, researchers writing in the American Journal of Medicine discovered something that should change how you think about medical news. They looked at 101 studies published in top scientific journals between 1979 and 1983 that claimed a new therapy or medical technology was very promising. Only five, they found out, made it to market within a decade. Only one (ACE inhibitors, a pharmaceutical drug) was still extensively used at the time of their publication.

One.

It’s a fact that all studies are biased and flawed in their own unique ways. The truth usually lies somewhere in a flurry of research on the same question. This means real insights don't come by way of miraculous, one-off findings or divinely ordained eureka moments; they happen after a long, plodding process of vetting and repeating tests, and peer-to-peer discussion. The aim is to make sure findings are accurate and not the result of a quirk in one experiment or the biased crusade of a lone researcher.

As science is working itself out, we reporters and our audiences seize on “promising findings.” It's exciting to hear about a brand new idea that maybe — just maybe — could revolutionize medicine and stop some scourge people suffer through. We're often prodded along by overhyping scientists like Zamboni, who are under their own pressure to attract research funding and publications.

We don't wait for scientific consensus; we report a little too early, and we lead patients and policymakers down wasteful, harmful, or redundant paths that end in dashed hope and failed medicine.

Read the rest here.