Background and notation for simple Bayes sampling example

Previously we considered maximum likelihood estimation for the location θ and scale s of a Cauchy model for the data

               1.2 2.4 1.3 1.3 0.0 1.0 1.8 0.8 4.6 1.4

I.e. the model is that y_{1}, …, y_{10} as above are modelled as iid according to

p(y_{i}|θ, s) = \frac{1}{πs} · \frac{1}{1 + (\frac{y_{i}−θ}{s} )^{2}}

and likelihood

L(\underline{y}|θ, s) = ∏_{i=1^{n}} p(y_{i}|θ, s)

We now consider a simple sampling scheme to explore the posterior distribution of the parameters θ and s in this model, when θ and s have independent priors

θ ∼ N(θ_{0}, σ_{0}^{2})
s^{2} ∼ 1/χ_{df}^{2}

First, we consider generating a sample from the posterior π(θ, s|\underline{y}) by the simple rejection scheme

Rejection for Bayes.

• Generate θ from π(θ);

• Generate U from Unif[0, L(y|θ̂)π(θ)];

• If U < L(y|θ)π(θ), accept θ as an observation from p(θ|y); otherwise reject θ and start over.

(in the example we drop θ from steps 2 and 3 and instead generate U ∼ Unif[0, 1] and check U < L(y|θ)/L(y|θ̂)).

Second, we can use importance sampling/SIR to generate a sample from a posterior using a different prior, as follows: Given an iid sample

θ_{1}, …, θ_{M} from p(θ|\underline{y}) ∼ L(\underline{y}|θ)π(θ),

if we resample

θ_{1}^{∗}, …, θ_{n}^{∗} as i.i.d. from θ_{1}, …, θ_{M}, with probabilities q_{1}, … q_{M}

where q_{i} ∝ π^{∗}(θ_{i})/π(θ_{i}) the result will be

θ_{1}^{∗}, …, θ_{n}^{∗} iid, ∼^{approx} p^{∗}(θ|\underline{y}) ∝ L(\underline{y}|θ)π^{∗}(θ).