Homework 2 SOLUTIONS

2. RPD Ex. 1.3(30)

\begin{aligned}Y_{i} & = & β_{0} + β_{1}X_{i} + ε_{i} \\ ε_{i} & ∼ & NID(0, α^{2}) \end{aligned}

a.  E(Y_{i}) = E(β_{0} + β_{1}X_{i} + ε_{i}) = β_{0} + β_{1}X_{i} + E(ε_{i}) = β_{0} + β_{1}X_{i}
b.  σ^{2}(Y_{i}) = E [(β_{0} + β_{1}X_{i} + ε_{i})^{2}] − E^{2}(β_{0} + β_{1}X_{i} + ε_{i})
= E [(β_{0} + β_{1}X_{i})^{2} + 2(β_{0} + β_{i}X_{i})ε_{1} + E_{i}^{2}] − (β_{0} + β_{1}X_{i})^{2}
= (β_{0} + β_{1}X_{i})^{2} + 0 + E(ε_{i}^{2}) − (β_{0} + β_{i}X_{i})
= σ^{2}
c. Cov(Y_{i}, Y_{i′}) = E [(β_{0} + β_{1}X_{i} + ε_{i} − β_{0} − β_{1}X_{i})(β_{0} + β_{1}X_{i′} + ε_{i′} − β_{0} − β_{i}X_{i′})]
= E(ε_{i′}ε_{i′}) = 0 , i ≠ i′

3.

a. Show: ∑ Ŷ_{i}e_{i} = 0
∑ Ŷ_{i}e_{i} = ∑ Ŷ_{i}(Y_{i} − Ŷ_{i}) = ∑(β̂_{0} + β̂_{i}X_{i})(Y_{i} − β̂ − β̂_{i}X_{i})

= β̂_{0} ∑ Y_{i} + β̂_{i} ∑ X_{i}Y_{i} − nβ̂_{0}^{2} − β̂_{1}β̂_{0} ∑ X_{i} − β̂_{1}β̂_{0} ∑ X_{i} − β̂_{1}^{2} ∑ X_{i}^{2} = β̂_{i} \underbrace{(∑ Y_{i} − nβ̂_{0} − β̂_{1} ∑ X_{i})} +β̂_{1} \underbrace{(∑ X_{i}Y_{i} − β̂_{0} ∑ X_{i} − β̂_{1} ∑ X_{i}^{2})}

Normal Equation 1  Normal Equation 2

= β̂_{0} · 0 + β̂_{1} · 0 = 0

b. Show: ∑ Y_{i} = ∑ Ŷ_{i}
∑ Y_{i} = ∑(\overline{Y} − β_{1}\overline{X} + β_{1}X_{i}) = n\overline{Y} − β_{1} ∑(\overline{X} − X_{i}) = n\overline{Y} QED
c. Show: S_{Y Y} = SSReg + RSS

\begin{aligned}<=> & ∑(Y_{i} − \overline{Y})^{2} = ∑(Ŷ_{i} − \overline{Y})^{2} & + ∑(Y_{i} − Ŷ_{i})^{2} \\ <=> & ∑ Y_{i}^{2} − 2\overline{Y} ∑ Y_{i} + n\overline{Y}^{2} & = ∑ Ŷ_{i}^{2} − 2\overline{Y} ∑ Ŷ_{i} + n\overline{Y}^{2} + ∑ Y_{i}^{2} \\ <=> & 0 & = 2 [∑ Ŷ_{i}^{2} − ∑ Y_{i}Ŷ_{i}] \\ <=> & 0 & = 2 [∑ Ŷ_{i}^{2} − ∑(Ŷ_{i} + e_{i})Ŷ_{i}] \\ <=> & 0 & = 2 [∑ Ŷ_{i}^{2} − ∑ Ŷ_{i}^{2} − ∑ Ŷ_{i}e_{i}] \\ <=> & 0 & = −2 ∑ Ŷ_{i}e_{i} \\ <=> & 0 & =  0  QED \end{aligned}

Homework 2

4.

a. Show: β̂_{1} = r_{xy} · \frac{S_{Y}}{S_{X}}
β̂_{1} = \frac{∑(X_{i}−\overline{X})(Y_{i}−\overline{Y})}{∑(X_{i}−\overline{X})^{2}} = \frac{S_{Y}}{S_{X}} · \begin{matrix}∑ & (\frac{X_{i}−\overline{X}}{S_{X}} ) & (\frac{Y_{i}−\overline{Y}}{S_{Y}} ) \\ ∑ & \frac{(X_{i}−\overline{X})^{2}}{S^{2}_{X}} \end{matrix} = \frac{S_{Y}}{S_{X}} · r_{XY}

b.  R^{2} = \frac{S_{XY}^{2}}{S_{XX}S_{Y Y}} = \frac{[∑(X_{i}−\overline{X})(Y_{i}−\overline{Y})]^{2}}{∑(X_{i}−\overline{X})^{2} ∑(Y_{i}−\overline{Y})^{2}} = \frac{_{\frac{1}{S^{2}_{X} S^{2}_{Y}}} ·[∑(X_{i}−\overline{X})(Y_{i}−\overline{Y})]^{2}}{_{\frac{1}{S^{2}_{X} S^{2}_{Y}}} ∑(X_{i}−\overline{X})∑(Y_{i}−\overline{Y})^{2}}

= \begin{matrix}[∑ \frac{X_{i}−\overline{X}}{S_{X}} · \frac{Y_{i}−\overline{Y}}{S_{Y}} ] & ^{2} \\ [ & ∑ & (\frac{X_{i}−\overline{X}}{Sx} ) & ^{2} & ∑ & (\frac{Y_{i}−\overline{Y}}{S_{Y}} ) & ^{2} & ] \end{matrix} = R_{XY}^{2}

* Other approach:

SSReg = ∑(Ŷ_{i} − \overline{Y})^{2} = ∑(β̂_{0} + β̂_{i}X_{i} − \overline{Y})^{2} = ∑(\overline{Y} − β̂_{1}\overline{X} + β̂_{1}X_{i} − \overline{Y})^{2}
= β̂_{1}^{2} ∑(X_{i} − \overline{X})^{2}
=> R^{2} = \frac{SSReg}{S_{Y Y}} = ^{\frac{β̂_{1}^{2} ∑(X_{i}−\overline{X})^{2}}{S_{Y Y}}} = ^{\frac{r_{XY}^{2} · \frac{S^{2}_{Y}}{S^{2}_{X}} ·∑(X_{i}−\overline{X})^{2}}{S_{Y Y}}} = r_{XY}^{2}

5. Regress Moral on Ethnic.

(a) > myfit _lm(Moral ~ Ethnic, data = angell)
> summary(myfit)
Call: lm(formula = Moral ~ Ethnic, data = angell)
Residuals:
     Min     1Q Median     3Q   Max
 -6.678 -2.61 0.2493 2.297 6.693
Coefficients:
                 Value Std. Error  t value Pr(>|t|)
(Intercept)  14.4236   0.8251     17.4815   0.0000
      Ethnic  -0.1028   0.0221     -4.6450   0.0000
Residual standard error: 2.926 on 41 degrees of freedom
Multiple R-Squared: 0.3448
F-statistic: 21.58 on 1 and 41 degrees of freedom, the p-value is 3.486e-05
Correlation of Coefficients:
        (Intercept)
Ethnic -0.8411
> plot(angell$Ethnic,angell$Moral)
> abline(myfit)

From the summary of the regression and from the scatter plot we see that Moral has a generally negative relationship with Ethnic. This is borne out by the estimated regression equation

ŷ = β̂_{0} + β̂_{1}x
14.4236 − 0.1028 · x

The residual standard error is listed directly in the summary printout as well,

s = \sqrt{RSS/(n − 2)} = 2.926

and the correlation between Ethnic and Moral is

r_{xy} = \sqrt{R^{2}} = \sqrt{0.3448} = −0.5872

(with the negative sign corresponding to the negative value of β̂_{1}).

(b) > plot(angell$Ethnic,residuals(myfit))
> abline(h=0) # or abline(0,0)
> qqenv(residuals(myfit))

The quantile normal plot (on the right) doesn’t show any severe violation of the assumption of normality for the residuals from the regression, and the raw residual plot (on the left) doesn’t show any particular pattern of positive and negative residuals, and no unusual features (outliers and the like). On this basis, the regression looks pretty good. However, the raw residual plot does show that the Ethnic variable is bunched up in lower values, with some right skewing.

6.

> attach(angell)
> par(mfrow=c(3,2))
> hist(Ethnic)
> qqenv(Ethnic)
> hist(sqrt(Ethnic))
> qqenv(sqrt(Ethnic))
> hist(log(Ethnic))
> qqenv(log(Ethnic))

Ethnic has quite a lot of right skew, as we suspected from the residual plot above. sqrtEthnic still has some right skew, though there are relatively few violations in the quantile-normal plot. log(Ethnic) mostly removes the right skew (and maybe even adds a little left skew), though we have not quite got a symmetric distribution yet. Still, among these three, log(Ethnic) is best.

7. > NewEthnic _log(Ethnic)
> newfit _lm(Moral ~ NewEthnic)
> summary(newfit)
Call: lm(formula = Moral ~ NewEthnic)
Residuals:
   Min      1Q  Median     3Q   Max
-6.242 -2.014 -0.2082 2.486 6.949
Coefficients:
                Value Std. Error  t value Pr(>|t|)
(Intercept)  22.7138   2.5478      8.9150   0.0000
 NewEthnic  -3.5246   0.7678     -4.5906   0.0000
Residual standard error: 2.938 on 41 degrees of freedom
Multiple R-Squared: 0.3395
F-statistic: 21.07 on 1 and 41 degrees of freedom, the p-value is 4.141e-05

Correlation of Coefficients:
           (Intercept)
NewEthnic -0.9844
> plot(NewEthnic,Moral)
> abline(lm(Moral ~ NewEthnic))
> plot(Ethnic,Moral)
> lines(sort(Ethnic),fitted(lm(Moral ~ NewEthnic))[order(Ethnic)])

I like the plot on the left (with the x-axis in the log(Ethnic) scale) for examining the regression, since I have some practice “eyeballing” a straight-line fit to data. The plot on the right, with the xaxis in the Ethnic scale, is more useful for discussion with a colleague, say, in sociology, since Ethnic is the variable of interest, not log(Ethnic). Moreover, the plot on the right clearly shows the steeper slope in the relationship between Ethnic and Moral at low values of Ethnic, which is totally obscured in the log(Ethnic) plot; this increase in slope for lower Ethnic values might be scientifically interesting to the sociologist. Here are the corresponding residual plots:

Since residual plots are mostly for assessing the goodness of the regression, I would be inclined to stick with the plot on the left–again, since it corresponds to a completely linear model, which I am happier using my intuition with.