Elicitation is a key task for subjectivist Bayesians. While skeptics
hold that it cannot (or perhaps should not) be done, in practice it
brings statisticians closer to their clients and subject-matter-expert
colleagues. This paper reviews the state-of-the-art, reflecting
the experience of statisticians informed by the fruits of a long line of
psychological research into how people represent uncertain information
cognitively, and how they respond to questions about that
information. In a discussion of the elicitation process, the first
issue to address is what it means for an elicitation to be successful,
i.e. what criteria should be employed? Our answer is that a successful
elicitation faithfully represents the opinion of the person being
elicited. It is not necessarily ``true'' in some objectivistic sense,
and cannot be judged that way. We see elicitation as simply part of the
process of statistical modeling. Indeed in a hierarchical model it is
ambiguous at which point the likelihood ends and the prior
begins. Thus the same kinds of judgment that inform statistical
modeling in general also inform elicitation of prior distributions.
The psychological literature suggests that people are prone to certain
heuristics and biases in how they respond to situations involving
uncertainty. As a result, some of the ways of asking questions about
uncertain quantities are preferable to others, and appear to be more
reliable. However data are lacking on exactly how well the various
methods work, because it is unclear, other than by asking using an
elicitation method, just what the person believes. Consequently one is
reduced to indirect means of assessing elicitation methods.
The tool-chest of methods is growing. Historically the first methods
involved choosing hyperparameters using conjugate prior families, at a
time when these were the only families for which posterior
distributions could be computed. Modern computational methods such as
Markov Chain Monte Carlo have freed elicitation from this
constraint. As a result there are now both parametric and
non-parametric methods available for low-dimensional problems. High
dimensional problems are probably best thought of as lacking another
hierarchical level, which has the effect of reducing the
as-yet-unelicited parameter space.
Special considerations apply to the elicitation of group
opinions. Informal methods, such as Delphi, encourage the participants
to discuss the issue in the hope of reaching consensus. Formal
methods, such as weighted averages or logarithmic opinion pools, each
have mathematical characteristics that are uncomfortable. Finally,
there is the question of what a group opinion even means, since it is
not necessarily the opinion of any participant.
Keywords: Bayesian, group decisions, heuristics and biases, prior
distributions, subjective probability