\documentstyle[12pt,bj,txfonts,psfig,fancyheadings]{article}

\begin{document}

\frenchspacing

\def\date{\today}

\begin{center}
\large\bf
36-303: Sampling Surveys and Society\\
Post-Stratification Challenges\\
\rm Brian Junker\\
132E Baker Hall\\
brian@stat.cmu.edu\\
\date
\end{center}

\pagestyle{fancy}
\lhead{\date}
\rhead{Post-Stratification Challenges}
\thispagestyle{empty}

\tableofcontents

\clearpage

\section{Post-Stratification}

\subsection{Basic Post-Stratification}

As part of survey data collection it is a good idea to get general
demographic information (e.g. in our surveys: sex, age, class, major,
hometown, etc.).  After data collection we compare the proportions in
each of these categories in our sample with the same proportions in
the population.
\begin{itemize}
\item If they agree, great.  
\item If the disagree, we may
re-weight the sample to make them agree
\[
    \mbox{\em weight} = (\mbox{\em population proportion})/(\mbox{\em
      sample proportion} ) 
\]
\end{itemize}
These categories are called ``post-strata'', and the weights are
called ``post-stratification weights''.

\subsection{Using the Weights}

To illustrate, we consider an SRS of 100 college students at a
fictional university, taken to find out how many hours they work on
class-related things.  The survey was done by email/website, and
unfortunately the response rate was only 20\%, so we have only 20
observations in our sample.  

Because so few people responded, we are worried about selection
effects in the sample: how are the 20 students who responded different
from the 80 people who did not, and how will this affect our
estimates?  A way to try to make the sample look more like the
population is to calculate post-stratification weights and use the
weights in all of our calculations.

The basic data is as follows:

\begin{center}
\begin{tabular}{@{}c@{\hspace*{4em}}c@{}}
\begin{tabular}{ccc}
Sex & College & Hrs/Wk \\
\hline
\hline
M & Eng & 28 \\
M & Eng & 29 \\
M & Eng & 23 \\
M & Eng & 35 \\
M & Eng & 29 \\
M & Eng & 30 \\
M & Eng & 34 \\
M & Eng & 31 \\
F & Eng & 30 \\
F & Eng & 31 \\
\hline
\end{tabular} 
&
\begin{tabular}{ccc}
Sex & College & Hrs/Wk \\
\hline
\hline
F & Eng & 36 \\
F & Eng & 33 \\
M & Lib & 27 \\
M & Lib & 28 \\
F & Lib & 29 \\
F & Lib & 30 \\
F & Lib & 28 \\
F & Lib & 28 \\
F & Lib & 32 \\
F & Lib & 30 \\
\hline
\end{tabular} 
\end{tabular}
\end{center}
We have collected some demographic data (male/female and
engineering/liberal-arts) on the students, and we find that in the
sample we have the following proportions in post-strata created by
crossing these categories:
\begin{center}
\begin{tabular}{c|ccc}
Sex & Eng & Lib \\
\cline{1-3}
M & 8 & 2 \\
F & 4 & 6 \\
\multicolumn{3}{l}{Total} & 20
\end{tabular}
\end{center}
On the other hand, in the population of all studetns at this
university the counts are
\begin{center}
\begin{tabular}{c|ccc}
Sex & Eng & Lib \\
\cline{1-3}
M & 617 & 380 \\
F & 450 & 551 \\
\multicolumn{3}{l}{Total} & 1998
\end{tabular}
\end{center}

To adjust the sample proportions so that they better reflect the
population proportions of each of these four categories, we create
post-stratification weights according to the formula:
\[
    \mbox{\em weight} = (\mbox{\em population proportion})/(\mbox{\em
      sample proportion} ) 
\]
In particular the weights are:
Post-strat. weights:
\begin{eqnarray*}
	w_{M,E}=(617/1998)/(8/20) = 0.77 && w_{M,L}=(380/1998)/(2/20) = 1.90 \\
	w_{F,E}=(450/1998)/(4/20) = 1.13 && w_{F,L}=(551/1998)/(6/20) = 0.92
\end{eqnarray*}
We attach these weights to each of the 20 observations in the original
SRS, obtaining
\begin{center}
\begin{tabular}{@{}c@{\hspace*{4em}}c@{}}
\begin{tabular}{cccc}
Sex & College & Hrs/Wk & Wgt\\
\hline
\hline
M & Eng & 28 & 0.77 \\
M & Eng & 29 & 0.77\\
M & Eng & 23 & 0.77\\
M & Eng & 35 & 0.77\\
M & Eng & 29 & 0.77\\
M & Eng & 30 & 0.77\\
M & Eng & 34 & 0.77\\
M & Eng & 31 & 0.77\\
F & Eng & 30 & 1.13\\
F & Eng & 31 & 1.13\\
\hline
\end{tabular} 
&
\begin{tabular}{cccc}
Sex & College & Hrs/Wk  & Wgt \\
\hline
\hline
F & Eng & 36 & 1.13\\
F & Eng & 33 & 1.13\\
M & Lib & 27 & 1.90\\
M & Lib & 28 & 1.90\\
F & Lib & 29 & 0.92\\
F & Lib & 30 & 0.92\\
F & Lib & 28 & 0.92\\
F & Lib & 28 & 0.92\\
F & Lib & 32 & 0.92\\
F & Lib & 30 & 0.92\\
\hline
\end{tabular} 
\end{tabular}
\end{center}
Now we can compare the unweighted mean from the SRS,
\[
	\overline y_{srs} = \frac{1}{20}\sum_{i=1}^{20} y_i = 30.05~,
\]
with the weighted mean using the post-stratification weights,
\[
	\overline y_w = \frac{\sum_i w_i y_i}{\sum_i w_i} = 29.91~.
\]
There isn't much difference in this case, but you can see that there
could be a big difference if the sample proportions are far different
from the population ones.

We also need to adjust our variance calculation to account for the
weights.  There are two commonly-used methods:
\begin{description}
\item[Taylor Series Method] This method is based on a Taylor Series
  approximation (known as the ``delta method'' to statisticians) that
  was discussed in class.  We calculate
\begin{eqnarray*}
	\Var_{TS}(\overline y_w) 
	&\approx &
	\frac{1}{\left(\sum_i w_i\right)^2}
	\left[
	\Var(\sum_i w_i y_i) - 2\overline y_w Cov(\sum_i w_i y_i, \sum_i w_i)
	+ (\overline y_w)^2Var(\sum_i w_i)
	\right] \\
        & = &  0.46 
\end{eqnarray*}
where $\bar y_w = 29.91$ as above, $\bar w= \frac1n\sum_i w_i = 1.00$,
$\bar{wy}= \frac1n\sum_i w_i y_i = 29.91$,
and [assuming the pairs $(w_i,y_i)$ are independent for different $i$'s],
\begin{eqnarray*}
    \Var(\sum_{i=1}^n w_i) 
	& \approx & n  \cdot \frac{1}{n-1}\sum_{i=1}^n (w_i - \bar w)^2 
	= 2.26
\\
	\Var(\sum_{i=1}^n  y_i w_i) 
	& \approx & n \cdot \frac{1}{n-1} \sum_{i=1}^n (w_i y_i - \bar{wy})^2
	= 1788.84
\\
	\Cov(\sum_{i=1}^n y_i w_i, \sum_{i=1}^n w_i) 
	& \approx& 
	n \cdot \frac{1}{n-1} \sum_{i=1}^n (w_i y_i - \bar{wy}) (w_i - \bar w)
	= 60.64
\end{eqnarray*}

\item[Jackknife Method] In this method we create $n$ replicate data
  sets of $n-1$ observations each, by deleting each one of the
  original observations in turn.  We recalculate the
  post-stratification weights for each replicate sample, and
  recalculate
\[
	\overline y_w^{(r)} = \frac{\sum_{i=1}^n w_i^{(r)} y_i^{(r)}}{\sum_{i=1}^n w_i^{(r)}}
\]
for the $r^{th}$ replicate sample (based on deleting the $r^{th}$
observation from the original data set).  In the case of our fictional
college work survey we get the following $y_w^{(r)}$'s:
\begin{quote}\tt
29.99382 29.94970 30.21439 29.68501 29.94970 29.90558 29.72912 29.86147
30.09879 30.02371 29.64834 29.87356 30.00619 29.81600 29.93868 29.88352
29.99383 29.99383 29.77321 29.88352
\end{quote}
These can be combined to create a new estimate of the average hours of
work
\[
	\bar y_{JK} ~ = ~ \frac1n\sum_{r=1}^n \bar y_w^{(r)}
	~ = ~ 29.91
\]
(which in most cases should be equal to the original $\bar y_w$, as it
is here), and a new estimates of the variance
\[
	\Var_{JK}(\bar y_w) 
	~ = ~ \frac{n-1}{n} \sum_{r=1}^n (\bar y_w^{(r)} - \bar y_{jk})^2
	~ = ~ 0.34
\]
(Note that we multiply the sum of squared differences by $(n-1)/n$
because the $y_w^{(r)}$'s are all positively correlated (they are
calculated from mostly the same data), rather than the usual $1/(n-1)$
which would be appropriate if the $y_w^{(r)}$'s were independent of
each other).

\end{description}
In most cases, $\Var_{TS}(\bar y_w)$ and $\Var_{JK}(\bar y_w)$ will be
slightly larger than $\Var(\bar y_{srs})$, reflecting extra
uncertainty in calculating the weights.

Once we have the variance, we can calculate confidence intervals,
etc., as usual, for example an approximate 95\% interval would be:
\[
	(\bar y_w - 2\sqrt{(1-n/N)Var(\bar y_w)}~,~~ 
	\bar y_w + 2\sqrt{(1-n/N)Var(\bar y_w)})
\]
For example, plugging in $\bar y_w=29.91$, $\Var_{TS}(\bar y_w) =
0.46$, $n=20$, and $N=1998$, we obtain the interval
\[
         (28.56~,~~31.26)
\]
for the overall average number of hours worked per week by students at
this university.

\subsection{Pro's \& Con's}

Post-stratification weights can fix
\begin{itemize}
\item Disproportionate sampling of post strata;
\item Disproportionate nonresponse across poststrata 
\end{itemize}
However, the weights we are calculating here, which simply adjust the
sample proportions so that they equal the population proportions, only
work if the sampling/nonresponse process is {\bf ignorable} within
post-strata.  That is, {\em nonresponse does not depend on the answer
  you would have gotten if the person had responded.}

If the sampling/nonresponse process is {\bf non-ignorable} then these
weights don't work; other weights have to be used.  These new weights
would be based on a model that you build, to explain why the
non-responders are not responding.  Weights for non-ignorable
non-response are only as good as your model for nonresponse.

These weights are a very big deal in pre-election phone surveys for
example (resp. rate 20--30\%, weights account for ignorable and
nonignorable nonresponse).


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{What to Do About Empty Strata}

As in the example above, you can ``cross'' several different
demographic variables to make post-strata.  However, you will quickly
run out of sample observations to fill each post-stratum, if you try
to combine very many variables, or variables with many categories,
this way.

For example, if you have collected data on CMU students for
\begin{itemize}
\item Class: Freshman, Sophomore, Junior, Senior, 5$^{th}$-year
\item School: HSS, CIT, MCS, CFA, Tepper, Heinz, CS
\end{itemize}
then you will have $5\times7=35$ post-strata.  If your sample size is
only around 100 students, you will very likely have some empty
post-strata in the sample, even though there are students in every
post-stratum in the population.

Here are two strategies to try, in this case:
\begin{description}
\item[Use fewer variables to post-stratify.] ~\\
  For example, maybe
  the sample proportions of class (Fr/So/Jr/etc) are a worse match to
  the population proportions than for school.  Then try to
  post-sratify just on class.  In many cases, this will also help
  correct proportions for school. 

\item[Build weights for each variable in succession.] ~\\
 For example,
\begin{itemize}
\item First build post-stratification weights for {\em class}, using
  the sample counts and population counts for class.
\item Then build post-stratification weights for {\em school}, using
  the sample counts and population counts for school.
\end{itemize}
Your final weights will be the product of these two sets of weights.  
\begin{quote}\em
Using the college work survey example above, we find
\[
   w_E = [(617+450)/1998]/[(8+4)/20] = 0.8901~,~~
   w_L = [(380+551)/1998]/[(2+6)/20] = 1.1649
\]
and
\[
   w_M = [(617+380)/1998]/[(8+2)/20] = 0.9980~,~~
   w_F = [(450+551)/1998]/[(4+6)/20] = 1.0020
\]
so the combined weights would be 
\begin{eqnarray*}
  w^*_{M,E}=(0.9980)(0.8901) = 0.8883 && w^*_{M,L}= (0.9980)(1.1649) = 1.1647\\
  w^*_{F,E}=(1.0020)(0.8901) = 0.8919 && w^*_{F,L}=(1.0020)(1.1649) = 1.1672
\end{eqnarray*}
These weights differ a bit from the original weights we calculated
\begin{eqnarray*}
	w_{M,E}=(617/1998)/(8/20) = 0.77 && w_{M,L}=(380/1998)/(2/20) = 1.90 \\
	w_{F,E}=(450/1998)/(4/20) = 1.13 && w_{F,L}=(551/1998)/(6/20) = 0.92
\end{eqnarray*}
because the $w^*$'s assume that sex and college are independent in the
sample and population, whereas the $w$'s do not assume independence.
\end{quote}
\end{description}
It is probably better to stick with the first strategy if you can.
Even professional surveys, such as the 2007 Pew survey on Religion in
America (that we looked at briefly in class a few weeks ago), uses
this method.  



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{How to Post-Stratify Within Sampling Strata}

\subsection{Sampling (Design) Strata}

Sampling strata (also called design strata or pre-strata) are strata
that you create before you do the survey.  For example, perhaps you
want to stratify on school within CMU, so you take a separate SRS
within each of CMU's seven schools.  The schools are {\em sampling
  strata}.

\subsection{Checking Post-Stratum Proportions Within Sampling Strata}

If you are worried about nonresponse within sampling strata, you can
also post-stratify within sampling strata.  For example, you could
post-stratify on class (Fr/So/Jr/etc.) within each sampling stratum. 

You can then check to see if the sampling proportions of each class
(post-stratum) match the population proportions, separately within
each school (sampling stratum).

\subsection{Post-Stratum Weights Within Stratified Sampling
  Calculations}

If you find that the post-stratum proportions do not match within
each sampling stratum, then you should create 
\begin{itemize}
\item Post-stratification weights separately within each stratum $h$;
\item $\bar y_{h,w}$'s separately within each stratum $h$;
\item $\Var(\bar y_{h,w})$'s separately within each stratum $h$.
\end{itemize}
These can then be combined using the usual formulas for stratified
sampling:
\[
  \bar y_{st,w} = \sum_{h=1}^H W_h \bar y_{h,w}
  = \sum_{h=1}^H \left(\frac{N_h}{N}\right) \bar y_{h,w}
\]
and
\[
  \Var(\overline y_{st,w}) = \sum_{h=1}^H  W_h^2
  (1- f_h) \Var(\bar y_{h,w})
  = \sum_{h=1}^H  \left(\frac{N_h}{N}\right)^2
  (1- n_h/N_h) \Var(\bar y_{h,w})
\]
where $\Var(\bar y_{h,w})$ can be calculated using the TS or JK
method, within each stratum $h$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\end{document}
