Jerry Adler in Pacific Standard [h/t: Lindsay Beyerstein] (Photo: Pacific Standard):
[E]xperimental science, it turns out, is no less susceptible to a good, thorough hoaxing than postmodern blather was.
The prank announced itself at the outset: In 2011, a psychologist named Joseph P. Simmons and two colleagues set out to use real experimental data to prove an impossible hypothesis. Not merely improbable or surprising, but downright ridiculous. The hypothesis: that listening to The Beatles’ “When I’m Sixty-Four” makes people younger. The method: Recruit a small sample of undergraduates to listen to either The Beatles song or one of two other tracks, then administer a questionnaire asking for a number of random and irrelevant facts and opinions—their parents’ ages, their restaurant preferences, the name of a Canadian football quarterback, and so on. The result: By strategically arranging their data and carefully wording their findings, the psychologists “proved” that randomly selected people who hear “When I’m Sixty-Four” are, in fact, younger than people who don’t.
The statistical sleight of hand involved in arriving at this result is a little complicated (more on this later), but the authors’ point was relatively simple. They wanted to draw attention to a glaring problem with modern scientific protocol: Between the laboratory and the published study lies a gap that must be bridged by the laborious process of data analysis. As Simmons and his co-authors showed, this process is a virtual black box that, as currently constructed, “allows presenting anything as significant.” And if you can prove anything you want from your data, what, if anything, do you really know?
More here.