Cathleen O’Grady in Science:
The 2012 paper, published in the Proceedings of the National Academy of Sciences (PNAS), reported a field study for which an unnamed insurance company purportedly randomized 13,488 customers to sign an honesty declaration at either the top or bottom of a form asking for an update to their odometer reading. Those who signed at the top were more honest, according to the study: They reported driving 2428 miles (3907 kilometers) more on average than those who signed at the bottom, which would result in a higher insurance premium. The paper also contained data from two lab experiments showing similar results from upfront honesty declarations.
The Obama administration’s Social and Behavioral Sciences Team recommended the intervention as a “nonfinancial incentive” to improve honesty, for instance on tax declarations, in its 2016 annual report. Lemonade, an insurance company, hired Ariely as its “chief behavioral officer.” But several other studies found that an upfront honesty declaration did not lead people to be more truthful; one even concluded it led to more false claims.
After discovering the result didn’t replicate in what he thought would be a “straightforward” extension study, one of the authors of the PNAS paper, Harvard Business School behavioral scientist Max Bazerman, asked the other authors to collaborate on a replication of one of their two lab experiments. This time, the team found no effects on honesty, it reported in 2020, again in PNAS.
While conducting the new lab study, Harvard Business School Ph.D. student Ariella Kristal found an odd detail in the original field study: Customers asked to sign at the top had significantly different baseline mileages—about 15,000 miles lower on average—than customers who signed at the bottom. The researchers reported this as a possible randomization failure in the 2020 paper, and also published the full data set.
Some time later, a group of anonymous researchers downloaded those data, according to last week’s post on Data Colada. A simple look at the participants’ mileage distribution revealed something very suspicious. Other data sets of people’s driving distances show a bell curve, with some people driving a lot, a few very little, and most somewhere in the middle. In the 2012 study, there was an unusually equal spread: Roughly the same number of people drove every distance between 0 and 50,000 miles. “I was flabbergasted,” says the researcher who made the discovery. (They spoke to Science on condition of anonymity because of fears for their career.)
Worrying that PNAS would not investigate the issue thoroughly, the whistleblower contacted the Data Colada bloggers instead, who conducted a follow-up review that convinced them the field study results were statistically impossible.
More here.