36-303: Sampling, Surveys and Society Exam 2 Tue Apr 12, 2011

• You have 80 minutes for this exam.

• The exam is closed-book, closed notes.

• A calculator is allowed.

• Two formula sheets are provided for your convenience.

• Please write all your answers on the exam itself; your work must be your own.

Question Points Possible Points Earned
1  20 FIX
2  20
3  18 FIX
4  24 FIX
5  18 FIX
Total  100

Name:

Signature:

Some Useful Formulas From the Statistics of Survey Sampling, I

Equally-Likely Outcomes & Counting

• If K outcomes O_{1}, …, O_{K} are equally likely, then the probability of any one of them is 1/K.

• Consider taking a sample of n objects from a population of N objects.

– Sampling with replacement, there are N^{n} possible samples of size n; the probability of any one of them is 1/N^{n}.

– Sampling without replacement, there are \binom{N}{n} = \frac{N!}{n!(N−n)!} possible samples of size n [where N! = N · (N − 1) · (N − 2) ⋯ 3 · 2 · 1], so the probability of any one of them is 1 \begin{matrix}/ \\ ( & {N \atop n} & ) \end{matrix}.

Discrete Random Variables Let X and Y be random variables with sample spaces {x_{1}, …, x_{K}} and {y_{1}, …, y_{K}} and distributions

P[X = x_{i}, Y = y_{j}] = p_{ij} , P[X = x_{i}] = p_{i·} = ∑_{j=1}^{K} p_{ij} , P[Y = y_{j}] = p_{·j} = ∑_{i=1}^{K} p_{ij}

Then, for example

E[X] = ∑_{i=1}^{K} x_{i}p_{i·} , Var(X) = ∑_{i=1}^{K} (x_{i} − E[X])^{2}p_{i·} , Cov(X, Y) = ∑_{i=1}^{K} (x_{i} − E[X])(y_{i} − E[Y])p_{ij}

P[X = x_{i}|Y = y_{j}] = p_{ij}/p_{·j} , E[X|Y = y_{j}] = ∑_{i=1}^{K} x_{i}P[X = x_{i}|Y = y_{j}] , E[aX + bY + c] = aE[X] + bE[Y] + c

Random Sampling From a Finite Population

Consider a population of size N and a sample of size n. Let y_{i} be the (fixed) values of some variable of interest in the population (such as a person’s age, or whether they would vote for Obama). Let

Z_{i} = \{ \begin{matrix}1, if i is in the sample \\ 0, else \end{matrix}

be the random sample inclusion indicators, and let Y_{i} be the random observations in the sample. Then the sample average can be written

\overline{Y} = \frac{1}{n} ∑_{i=1}^{n} Y_{i} = \frac{1}{n} ∑_{i=1}^{N} Z_{i}y_{i}

The Z_{i}’s are Bernoulli random variables with

E[Z_{i}] = \frac{n}{N} , Var(Z_{i}) = \frac{n}{N} (1 − \frac{n}{N}) , Cov(Z_{i}, Z_{j}) = −\frac{1}{N − 1} \frac{n}{N} (1 − \frac{n}{N})

Confidence Intervals and Sample Size

(a) A CLT-based 100(1 − α)% confidence interval for the population mean is (\overline{Y}− z_{α/2}SE , \overline{Y}+ z_{α/2}SE).

(b) For sampling with replacement from an infinite population, SE = SD/ \sqrt{n}.

(c) For sampling without replacement from a finite population, the SE has to be multiplied by the finite population correction (FPC).

(d) For a given margin of error (ME, half the width of the CI) and confidence level 1 − α, we can find the sample

size by solving

z_{α/2}SE < ME

for n. The same approach works for both SRS with replacement (using the SE in (b)) and SRS without replacement (using the SE in (c)).

Some Useful Formulas From the Statistics of Survey Sampling, II

Stratified Sampling Consider H strata with population counts N = ∑ _{h=1}^{H} N_{h} and sample counts n = ∑ _{h=1}^{H} n_{h}. Let f_{h} = n_{h}/N_{h}; W_{h} = N_{h}/N; and \overline{y}_{h} = \frac{1}{n_{h}} ∑ _{i=1}^{n_{h}} y_{ih} in each stratum, and let s_{h}^{2} = \frac{1}{n_{h}−1} ∑ _{i}(y_{ih} − \overline{y}_{h})^{2} be the sample variance in each stratum. Then

\overline{y}_{st} = ∑_{h=1}^{H} W_{h}\overline{y}_{h} , Var(\overline{y}_{st}) ≈ ∑_{h=1}^{H} W_{h}^{2}(1 − f_{h})\frac{s_{h}^{2}}{n_{h}} , DEFF = \frac{Var(\overline{y}_{st})}{Var(\overline{y}_{srs})} = \frac{∑ _{h=1}^{H} W_{h}^{2}(1 − f_{h})\frac{s_{h}^{2}}{n_{h}}}{(1 − f)\frac{s^{2}}{n}}

Cluster Sampling Consider a population of N clusters. We take an SRS S of n clusters, and all units within each sampled cluster (onestage clustering). Assume clusters all have same size M. Let \overline{y}_{i} = \frac{1}{M} ∑ _{j=1}^{M} y_{ij} in each cluster. Then

\overline{y}_{cl} = \frac{1}{n} ∑_{i∈S} \overline{y}_{i} , Var(\overline{y}_{cl}) ≈ (1 − \frac{n}{N}) \frac{1}{n}s_{\overline{y}_{i}}^{2} = (1 − \frac{n}{N}) \frac{1}{n} [\frac{1}{n − 1} ∑_{i∈S} (\overline{y}_{i} − \overline{y}_{cl})^{2}]

and

DEFF = \frac{Var(\overline{y}_{cl})}{Var(\overline{y}_{srs})} = \frac{Ms_{\overline{y}_{i}}^{2}}{s_{y_{ij}}^{2}} ≈ 1 + (M − 1)ρ

where s_{\overline{y}_{i}}^{2} is the sample varance of the cluster means, s_{y_{ij}}^{2} is the sample variance of the individual observations, and ρ is the intraclass (intracluster) correlation, or ICC. Post-Stratification Weights and Means As part of survey data collection it is a good idea to get general demographic information (e.g. in our surveys: sex, age, class, major, hometown, etc.). After data collection we compare the proportions in each of these categories in our sample with the same proportions in the population. If they agree, great. If not, calculate

w_{i} = (N_{h}/N)/(n_{h}/n) for each i in post-stratum h , and \overline{y}_{w} = \frac{∑ _{i} w_{i}y_{i}}{∑ _{i} w_{i}}

Post-Stratification Variance Calculations \underline{Taylor series}:

Var_{TS}(\overline{y}_{w}) ≈ \frac{1}{(∑ _{i} w_{i})_{2}} [Var(∑_{i} w_{i}y_{i}) − 2\overline{y}_{w}Cov(∑_{i} w_{i}y_{i}, ∑_{i} w_{i}) + (\overline{y}_{w})^{2}Var(∑_{i} w_{i})]

where \overline{y}_{w} is as above, \overline{w} = \frac{1}{n} ∑ _{i} w_{i}, \overline{wy}= \frac{1}{n} ∑ _{i} w_{i}y_{i},

Var(∑ _{i=1}^{n} w_{i}) ≈ n · \frac{1}{n−1} ∑ _{i=1}^{n}(w_{i} − \overline{w})^{2}, Var(∑ _{i=1}^{n} y_{i}w_{i}) ≈ n · \frac{1}{n−1} ∑ _{i=1}^{n}(w_{i}y_{i} − \overline{wy})^{2},
Cov(∑ _{i=1}^{n} y_{i}w_{i}, ∑ _{i=1}^{n} w_{i}) ≈ n · \frac{1}{n−1} ∑ _{i=1}^{n}(w_{i}y_{i} − \overline{wy})(w_{i} − \overline{w})

\underline{Jackknife}:

• Replicate n times (by removing one obs. each time and recalculating weights):

\overline{y}_{w}^{(r)} = \frac{∑ _{i=1}^{n} w_{i}^{(r)}y_{i}^{(r)}}{∑ _{i=1}^{n} w_{i}^{(r)}}

• Calculate

\overline{y}_{JK} = \frac{1}{n} ∑_{r=1}^{n} \overline{y}_{w}^{(r)} , Var_{JK}(\overline{y}_{w}) ≈ \frac{n − 1}{n} ∑_{r=1}^{n} (\overline{y}_{w}^{(r)} − \overline{y}_{jk})^{2}

April 12, 2011 Name:

1. [20 pts] FIX. Multiple Choice (4 parts). For each part, circle the roman numeral of the one best answer.

(a) [5 pts] Let Y_{i} be the number of monthly neighborhood watch meetings attended by the i^{th} resident in a neighborhood, in the last year. You are going to conduct a survey, using a SRS to estimate \overline{Y}_{pop}, the population mean number of meetings attended by neighborhood residents. Among the N_{R} residents who would respond to your survey, the mean number of meetings is \overline{Y}_{R}, and among the N_{M} number of residents who would not respond, the mean number of meetings attended is \overline{Y}_{M}. In class we showed that the bias between \overline{Y}_{R} and \overline{Y}_{pop}, due to missing responses, is

\overline{Y}_{R}− \overline{Y}_{pop} = \frac{N_{M}}{N}(\overline{Y}_{R}− \overline{Y}_{M})

where N = N_{R}+ N_{M}.

Which statement below is false (or, circle iv. if all are OK)?

i. All other things being equal, the more people who would not respond, the smaller the bias

due to missing responses. ii. All other things being equal, the bigger the difference between mean number of meetings

attended by nonresponders, vs the mean number attended by responders, the bigger the bias due to missing responses.

iii. The larger your SRS, the better you can estimate \overline{Y}_{R}. iv. All of the above statements are true.

(b) [5 pts] In one-stage clustered sampling, the ICC ρ measures

i. The correlation between observations in different clusters. ii. The correlation between the cluster means of different clusters. iii. The correlation between observations in the same cluster. iv. The correlation between the cluster mean and the individual observations in the cluster.

(c) [5 pts] A 36-303 survey project constructs an SRS of undergraduates and directs them to a website where they can answer the survey questions. However some respondents are never presented with the last page of the survey, due to a bug in the website software (the error can happen at any time to any respondent, but only a few respondents fail to complete the survey for this reason). This is most likely an example of

i. Unit nonresponse, Missing At Random (MAR) ii. Item nonresponse, Missing Completely At Random (MCAR) iii. Unit nonresponse, Missing Completely At Random (MCAR) iv. Item nonresponse, Missing At Random (MAR)

(d) [5 pts] Suppose we divide a sampling frame into groups, which we may treat as either strata for stratified sampling, or clusters for cluster sampling. If we make the groups so that observations within groups are more similar to each other, and observations between groups are more different from each other, then, all other things being equal, we expect

i. The variance of the stratified sample mean \overline{y}_{st} will go up and the variance of the cluster

sample mean \overline{y}_{cl} will go down. ii. The variance of the stratified sample mean \overline{y}_{st} will go down and the variance of the cluster

sample mean \overline{y}_{cl} will go up. iii. Both variances will go up. iv. Both variances will go down.

[This space intentionally left blank] 2. [20 pts] Cluster sampling. A survey is conducted to find out the proportion of cell phone

users in a certain city. From a population of 2500 residential blocks, 10 are sampled at random without replacement, and each person residing on that block is asked whether they use a cell phone. We will assume there are exactly 40 people living in each block^{1}. This yields the following table of data:

Total # of  # of Cell  Proportion of Cell
Block h People M Phone Users c_{h} Phone Users p_{h}
1  40  10  0.25
2  40  8  0.20
3  40  16  0.40
4  40  15  0.38
5  40  24  0.60
6  40  17  0.42
7  40  12  0.30
8  40  13  0.32
9  40  16  0.40
10  40  13  0.32
Total  400  144

(a) [4 pts] This is an example of one-stage clustered sampling. Circle one of each pair of boldface choices: The primary sampling units (psu’s) are the (blocks, residents), and the secondary sampling units (ssu’s) are the (blocks, residents).

(b) [4 pts] Ignoring the clustering and treating this as an SRS of 400 residents, estimate the proportion of cell phone users and its standard error.

[continued on next page] (c) [4 pts] Now re-estimate the proportion of cell phone users and its standard error, using

1This is a reasonable approximation as long as all the blocks have close to 40 residents.

appropriate cluster sampling methods. Hint: to reduce calculation, use the fact that in the table above, s_{p_{h}}^{2} = 0.1214333.

(d) [4 pts] Calculate the design effect DEFF for this design.

(e) [4 pts] Calculate ρ, the correlation between responses from residents on the same block.

3. [18 pts] FIX Approval Rating of Sen. Diane Feinstein, Part I (3 parts).

In late February 2009, the Field Research Corporation (http://www.field.com/fieldpollonline) conducted a poll of Democrats and Republicans in California to assess voters’ approval of US Sentator Diane Feinstein, and judge whether any potential candidates might be able to unseat her in the midterm elections in November 2010.

Approximately equal numbers of Democrats (342) and Republicans (298) were contacted by phone; 267 of the Democrats and 98 of the Republicans approved of the job Sen. Feinstein was doing.

According to the Ballot Access News (http://www.ballot-access.org/2008/120108.html), there were 7,683,495 registered Democrats and 5,428,052 registered Republicans in California at approximately the same time as the Field Poll was being taken.

(a) [6 pts] Treating the sample of Democrats as an SRS w/o replacement, compute p̂_{dem}, the proportion of Democrats who approve of Sen. Feinstein, and its standard error, SE(p̂_{dem}).

(b) [6 pts] Do the same for Republicans: compute p̂_{rep} and SE(p̂_{rep}).

(c) [6 pts] A confidence interval for the difference between the true proportions of Democrats and Republicans who approve of Senator Feinstein can be computed with the help of the pooled SE, \sqrt{SE(p̂_{dem})^{2} + SE(p̂_{rep})^{2}}. Compute this pooled SE and give an approximate 95% confidence interval for this difference in proportions of voters who approve of Sen. Feinstein.

4. [24 pts] FIX Response rates and missing data. You are completing a telephone survey of an SRS of 1000 members of a much larger professional organization, regarding their level of involvement in support of the organization. Currently the response rate is 80%, with 52.5% of those responding saying they attend every monthly meeting of the local chapter of the organization, and 47.5% saying they do not.

(a) [ pts] Does it seem likely that the 20% who did not respond to your survey are missing completely at random (MCAR, ignorable missingness) or missing not at random (MNAR, non-ignorable missingness)? Choose one and explain your reasoning.

(b) [ pts] Which of the following is more likely to be correct (circle \underline{one}):

• An unbiased estimate of the population proportion that attends every meeting is 52.5%

• An unbiased estimate of the population proportion that attends every meeting is probably less than 52.5%.

• An unbiased estimate of the population proportion that attends every meeting is probably more than 52.5%

(c) [ pts] To get the sample size nearer to the target of n = 1000, you could either

• Ask the organization to pay for you to call a new SRS of size 250, hoping that (0.80)(250) = 200 people choose to respond (cost: $400 because it just involves routine telephone survey operations); or

• Ask the organization to pay for you to followup with the 200 in your original sample who didn’t respond yet, to try to get their responses (cost: $1600 because it involves individual attention to each non-respondent).

Which do you choose, and why?

5. [18 pts] FIX Data Collection Methods (3 parts.) In each situation below, choose the best mode of data collection, and give a reason why. For the “why” question, write complete, clear sentences. Your response does not need to fill all of the available space^{2}. Strong reasoning and clear writing count much more than length.

Think about tradeoffs between feasibility, cost, response rate & nonresponse bias, availability of information on which respondents can base their answers, etc. You may assume there is a good sampling frame, and that a strong sampling method like SRS, stratified or clustered sampling, will be used.

(a) [6 pts] Your market research firm has been asked to conduct a survey in Pittsburgh to see who would purchase a new, sweet, fruity carbonated soft drink.

• Which mode of data collection (circle the best answer)? i. Telephone survey. ii. Face to Face interviews. iii. Internet (web-based) survey.

• Why?

2On the other hand, if you need more space, use the back of the page.

(b) [6 pts] The Governor of your state has asked you to design a survey to assess the views of residents of the state toward public transportation and to evaluate how likely they would be to use alternative modes of transportation.

• Which mode of data collection (circle the best answer)? i. Telephone survey. ii. Face to Face interviews. iii. Internet (web-based) survey.

• Why?

(c) [6 pts] The American Statsitical Association wishes to do a survey of its members across the United States on their views toward certification of statisticians (statisticians would need pass a test in order to get a license to practice statistics, like doctors, some engineering professions, etc.).

• Which mode of data collection (circle the best answer)? i. Telephone survey. ii. Face to Face interviews. iii. Internet (web-based) survey.

• Why?