Tag Archives: post-hoc analysis

Douglas Hubbard

James Randi, retired magician and renowned skeptic, set up this foundation for investigating paranormal claims scientifically. (He advised Emily on some issues of experimental protocol.) Randi created the $1 million “Randi Prize” for anyone who can scientifically prove extrasensory perception (ESP), clairvoyance, dowsing, and the like. Randi dislikes labeling his efforts as “debunking” paranormal claims since he just assesses the claim with scientific objectivity. But since hundreds of applicants have been un- able to claim the prize by passing simple scientific tests of their paranormal claims, debunking has been the net effect. Even before Emily’s experiment was published, Randi was also interested in therapeutic touch and was trying to test it. But, unlike Emily, he managed to recruit only one therapist who would agree to an objective test—and that person failed.

After these results were published, therapeutic touch proponents stated a variety of objections to the experimental method, claiming it proved nothing. Some stated that the distance of the energy field was really one to three inches, not the four or five inches Emily used in her experiment. Others stated that the energy field was fluid, not static, and Emily’s unmoving hand was an unfair test (despite the fact that patients usually lie still during their “treatment”). None of this surprises Randi. “People always have excuses afterward,” he says. “But prior to the experiment every one of the therapists were asked if they agreed with the conditions of the experiment. Not only did they agree, but they felt confident they would do well.” Of course, the best refutation of Emily’s results would simply be to set up a controlled, valid experiment that conclusively proves therapeutic touch does work. No such refutation has yet been offered.

Randi has run into retroactive excuses to explain failures to demonstrate paranormal skills so often that he has added another small demonstration to his tests. Prior to taking the test, Randi has subjects sign an affidavit stating that they agreed to the conditions of the test, that they would later offer no objections to the test, and that, in fact, they expected to do well under the stated conditions. At that point Randi hands them a sealed envelope. After the test, when they attempt to reject the outcome as poor experimental design, he asks them to open the envelope. The letter in the envelope simply states “You have agreed that the conditions were optimum and that you would offer no excuses after the test. You have now offered those excuses.”

Douglas Hubbard, How to Measure Anything: Finding the Value of “Intangibles” in Business, Hoboken, 2012, 2nd ed., pp. 15-16

Daniel Gilbert

There are many different techniques for collecting, interpreting, and analyzing facts, and different techniques often lead to different conclusions, which is why scientists disagree about the dangers of global warming, the benefits of supply-side economics, and the wisdom of low-carbohydrate diets. Good scientists deal with this complication by choosing the techniques they consider most appropriate and then accepting the conclusions that these techniques produce, regardless of what those conclusions might be. But bad scientists take advantage of this complication by choosing techniques that are especially likely to produce the conclusions they favour, thus allowing them to reach favoured conclusions by way of supportive facts. Decades of research suggests that when it comes to collecting and analyzing facts about ourselves and our experiences, most of us have the equivalent of an advanced degree in Really Bad Science.

Daniel Gilbert, Stumbling on Happiness, New York, 2005, p. 164