\documentstyle[12pt,bj,times]{article}

\begin{document}

\parskip=0.6em
\parindent=0pt

\let\ul\underline

\pagestyle{empty}

\begin{center}\large\bf
Background and notation for simple Bayes sampling example
\end{center}

Previously we considered maximum likelihood estimation for the
location $\theta$ and scale $s$ of a Cauchy model for the data

\hfil\verb|1.2 2.4 1.3 1.3 0.0 1.0 1.8 0.8 4.6 1.4|

I.e. the model is that $y_1, \ldots, y_{10}$ as above are modelled as
iid according to
\[
           p(y_i|\theta,s)
   = \frac{1}{\pi s} \cdot \frac{1}{1+\left(\frac{y_i-\theta}{s}\right)^2}
\]
and likelihood
\[
        L(\ul y|\theta,s) = \prod_{i=1^n} p(y_i|\theta,s)
\]
We now consider a simple sampling scheme to explore the posterior
distribution of the parameters $\theta$ and $s$ in this model, when
$\theta$ and $s$ have independent priors
\begin{eqnarray*}
  \theta & \sim & N(\theta_0,\sigma^2_0) \\
  s^2      & \sim & 1/\chi^2_{df}
\end{eqnarray*}

First, we consider generating a sample from the posterior
$\pi(\theta,s|\ul y)$ by the simple rejection scheme
\begin{quote}\frenchspacing
\bf Rejection for Bayes. \rm\em
\begin{itemize}\parsep=0pt\parskip=0pt\itemsep=0pt
\item Generate $\theta$ from $\pi(\theta)$;
\item Generate $U$ from $Unif[0,L(y|\hat\theta)\pi(\theta)]$;
\item If $U<L(y|\theta)\pi(\theta)$, accept $\theta$ as an observation
from $p(\theta|y)$; \\
\phantom{If $U<f(X)$,} otherwise reject $\theta$ and start over.
\end{itemize}
\end{quote}
(in the example we drop $\theta$ from steps 2 and 3 and instead
generate $U\sim Unif[0,1]$ and check $U<L(y|\theta)/L(y|\hat\theta)$).

Second, we can use importance sampling/SIR to generate a sample from a
posterior using a different prior, as follows:  Given an iid sample
\begin{center}
$\theta_1, \ldots, \theta_M$ from $p(\theta|\ul y) \sim L(\ul
y|\theta)\pi(\theta)$, 
\end{center}
if we resample
\begin{center}
  $\theta^*_1, \ldots, \theta^*_n$ as i.i.d. from $\theta_1,
  \ldots, \theta_M$, with probabilities $q_1, \ldots q_M$
\end{center}
where $q_i \propto \pi^*(\theta_i)/\pi(\theta_i)$
the result will be
\begin{center}
$\theta^*_1, \ldots, \theta^*_n$ iid, $\stackrel{approx}{\sim}$
$p^*(\theta|\ul y) \propto L(\ul
y|\theta)\pi^*(\theta)$.
\end{center}


\end{document}