How Incoherent is Fixed-Level Testing?

Mark J. Schervish, Teddy Seidenfeld and Joseph B. Kadane


It has long been known that the practice of testing all hypotheses at the same level (such as 0.05), regardless of the distribution of the data, is not consistent with Bayesian expected utility maximization. According to deFinetti's ``Dutch Book'' argument, procedures that are not consistent with expected utility maximization are incoherent and they lead to gambles that are sure to lose no matter what happens. In this paper, we propose a method to measure the degree to which incoherent procedures are sure to lose, so that we can distinguish slightly incoherent procedures from grossly incoherent ones. We present a detailed analysis of testing a simple hypothesis against a simple alternative as a case-study of how the method can work.

Keywords: Bookie; Brier score; Coherence; Escrow; Gambler; Hypothesis testing.

Heidi Sestrich
Here is the full postscript text for this technical report. It is 557540 bytes long.