Department of Statistics Unitmark
Dietrich College of Humanities and Social Sciences

How Incoherent is Fixed-Level Testing?

Publication Date

October, 1999

Publication Type

Tech Report

Author(s)

Mark J. Schervish, Teddy Seidenfeld and Joseph B. Kadane

Abstract

It has long been known that the practice of testing all hypotheses at the same level (such as 0.05), regardless of the distribution of the data, is not consistent with Bayesian expected utility maximization. According to deFinetti's "Dutch Book" argument, procedures that are not consistent with expected utility maximization are incoherent and they lead to gambles that are sure to lose no matter what happens. In this paper, we propose a method to measure the degree to which incoherent procedures are sure to lose, so that we can distinguish slightly incoherent procedures from grossly incoherent ones. We present a detailed analysis of testing a simple hypothesis against a simple alternative as a case-study of how the method can work.