Jason Mitchell at Harvard:
When an experiment is expected to yield certain results, and yet it fails to do so, scientists typically work to locate the source of the failure. In principle, the cause of an experimental failure could lurk anywhere; philosophers have pointed out that a failed experiment might very well indicate a previously undetected flaw in our system of logic and mathematics[1]. In practice, however, most scientists work from a mental checklist of likely culprits. At the top of this list, typically, are “nuts-and-bolts” details about the way in which the experiment was carried out—was my apparatus working properly? did my task operationalize the variable I was aiming for? did I carry out the appropriate statistical analysis in the correct way? and so on. Very often, the source of the failure is located here, if only because the list of practical mistakes that can undermine an experiment is so vast.
Considerably lower down the list are various doubts for expecting particular results in the first place, such as uncertainty about the theory that predicted them or skepticism about reports of similar effects[2]. In other words, when an experiment fails, scientists typically first assume that they bungled the details of the experiment before concluding that something must be wrong with their initial reasons for having conducted it in the first place (or that logic and mathematics suffer some fatal flaw). This makes good sense: it would be inane to discard an entire theoretical edifice because of one researcher’s undetected copy-and-paste error or other such practical oversight.
More here. And see also this comment from Neuroskeptic.
