\documentstyle[11pt,bj]{article}

\begin{document}

\def\eq#1{(\ref{#1})}

\section{Regression Equations for ANOVA; Reparametrization}

We have seen that the ``dummy variables'' linear regression version of
an ANOVA model is always overparametrized.  To see how ANOVA
reparametrizations overcome this, we now examine the one-way ANOVA
model from a linear regression point of view.

For simplicity, consider a one-way ANOVA problem for a factor with
three levels, $a$, $b$ and $c$, and two observations per cell.  Let 
\[
X_1 = \left\{\begin{array}{rr}1 & \mbox{at level $a$} \\
                              0 & \mbox{else} \end{array}\right.
~~~~~~~~~~~~
X_2 = \left\{\begin{array}{rr}1 & \mbox{at level $b$} \\
                              0 & \mbox{else} \end{array}\right.
~~~~~~~~~~~~
X_3 = \left\{\begin{array}{rr}1 & \mbox{at level $c$} \\
                              0 & \mbox{else} \end{array}\right.
\]
so the linear regression model is
\begin{equation}
        y_{ij} = \theta_0 + X_{1j}\theta_1 + X_{2j}\theta_2 
         + X_{3j}\theta_3 + \epsilon_{ij}
        \label{basic-1}
\end{equation}
or in matrix form
\[
        y = 
        \left[\begin{array}{c}
        y_{11} \\ y_{12} \\ y_{21} \\ y_{22} \\ y_{31} \\ y_{32}
        \end{array}\right]
        =
        X \theta + \epsilon
        =
        \left[\begin{array}{cccc}
        1 & 1 & 0 & 0 \\
        1 & 1 & 0 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 0 & 1 \\
        1 & 0 & 0 & 1 
        \end{array}\right]
        \left[\begin{array}{c}
        \theta_0 \\ \theta_1 \\ \theta_2 \\ \theta_3 
        \end{array}\right]
        +
        \left[\begin{array}{c}
        \epsilon_{11} \\ \epsilon_{12} \\ \epsilon_{21} \\ 
        \epsilon_{22} \\ \epsilon_{31} \\ \epsilon_{32}
        \end{array}\right]
\]
Basically we want to know how to introduce a linear constraint, to get
a new set of parameters that can be estimated uniquely (technical term:
the new parameters are {\em identifiable}).

\subsection{Matrix algebra details}

{\bf Note: \em This is a bit more mathematically formal than the usual
development of introducing constraints to make the parameters
identifiable, but I wanted to be able to explain the ``Helmert
parametrization'' in SPLUS, and this seems to require fooling around
like this.}

We want to consider reparametrizations of the model $y=X\theta +
\epsilon$, that is, we want to consider new sets of parameters
\begin{equation}
        \left[\begin{array}{c}
        \beta_1 \\ \beta_2 \\ \beta_3 
        \end{array}\right]
        =
        C \cdot
        \left[\begin{array}{c}
        \theta_1 \\ \theta_2 \\ \theta_3 \\ \theta_4
        \end{array}\right]
        \label{reparam}
\end{equation}
where 
\[
        C = \left[\begin{array}{cccc}
        c_{11} & c_{12} & c_{13} & c_{14} \\
        c_{21} & c_{22} & c_{23} & c_{24} \\
        c_{31} & c_{32} & c_{33} & c_{34} 
        \end{array}\right] .
\]

Our goal is to choose a new design matrix $R$ so that
\[
        y = X\theta + \epsilon = Z\beta + \epsilon
\]
i.e. the two regression problems are exactly the same (more formally:
$X$ and $Z$ have the same column space).  Note that, by \eq{reparam},
$R\beta = ZC\theta$, so it must be that
\[
        X ~ = ~ ZC .
\]
Now, we would like to solve this by ``multiplying on the right by
$C^{-1}$'', but unfortunately $C$ is not square, so it doesn't have an
inverse.  Instead, notice what happens if we multiply on the right by
$C'(CC')^{-1}$ (this is called a ``generalized inverse''):
\begin{eqnarray*}
        X & = & ZC \\
        X\cdot C'(CC')^{-1} & = & ZC\cdot C'(CC')^{-1} \\
        X\cdot C'(CC')^{-1} & = & Z 
\end{eqnarray*}
so now we have a way of finding $Z$ from $X$ and $C$, namely $Z = X
C'(CC')^{-1}$.

To summarize, our goal in understanding reparametrization is:

\hfil\fbox{\parbox{5in}{
\begin{description}
\item[Step 1.] Figure out what ``constraint(s)'' are being imposed on the
$\theta$'s to get rid of the linear dependency;
\item[Step 2.] Figure out how to write that as something like $\beta
= C\theta$;
\item[Step 3.] Compute the design matrix for the $\beta$'s as $Z =
XC'(CC')^{-1}$.
\item[Step 4.] The new regression is then $y = Z\beta + \epsilon$.
\end{description}
}}

Two notes before going on:
\begin{itemize}
\item In many simple problems, you can intuitively get from {\bf Step
1} to {\bf Step 4} without fooling around with the $C$ matrix in {\bf
Steps 2} and {\bf 3}.
\item After {\bf Step 4} it is often helpful to write the columns of
$Z$ in terms of the 0/1 dummy variables $X_1$, $X_2$, $X_3$, and
re-expressing the cell means $\mu_1$, $\mu_2$ and $\mu_3$ in terms of
the new parameters.
\end{itemize}

\subsection{Some illustrations}

\paragraph{Cell means model (delete the first column).} This is easy.
Setting $\theta_0=0$ is the same as deleting the first column in the
$X$ matrix; the matrix form of the regression becomes

this becomes
\[
        \left[\begin{array}{c}
        y_{11} \\ y_{12} \\ y_{21} \\ y_{22} \\ y_{31} \\ y_{32}
        \end{array}\right]
        =
        \left[\begin{array}{ccc}
        1 & 0 & 0 \\
        1 & 0 & 0 \\
        0 & 1 & 0 \\
        0 & 1 & 0 \\
        0 & 0 & 1 \\
        0 & 0 & 1 
        \end{array}\right]
        \left[\begin{array}{c}
        \theta_1 \\ \theta_2 \\ \theta_3 
        \end{array}\right]
        +
        \left[\begin{array}{c}
        \epsilon_{11} \\ \epsilon_{12} \\ \epsilon_{21} \\ 
        \epsilon_{22} \\ \epsilon_{31} \\ \epsilon_{32}
        \end{array}\right]
\]
and \eq{basic-1} becomes
\[
        y_{ij} = X_{1j}\theta_1 + X_{2j}\theta_2 
         + X_{3j}\theta_3 + \epsilon_{ij}
\]
and it is easy to see in the resulting model, $\mu_1 = \theta_1$,
$\mu_2 = \theta_2$ and $\mu_3 = \theta_3$, the cell means model.

Let's see how we would do this by setting up the $C$ matrix and
calculating $Z = XC'(CC')^{-1}$:

We want $\theta_0=0$ and
\begin{eqnarray*}
        \mu_1 & = & \theta_1 \\
        \mu_2 & = & \theta_2 \\
        \mu_3 & = & \theta_3
\end{eqnarray*}
or 
\[
        \left[\begin{array}{c}
        \mu_1 \\ \mu_2 \\ \mu_3 
        \end{array}\right]
        = 
        \left[\begin{array}{cccc}
        0 & 1 & 0 & 0 \\
        0 & 0 & 1 & 0 \\
        0 & 0 & 0 & 1
        \end{array}\right]
        \left[\begin{array}{c}
        \theta_0 \\ \theta_1 \\ \theta_2 \\ \theta_3 
        \end{array}\right]
        = C\theta
\]
Hence
\[
        Z = X C'(CC')^{-1} = 
        \left[\begin{array}{cccc}
        1 & 1 & 0 & 0 \\
        1 & 1 & 0 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 0 & 1 \\
        1 & 0 & 0 & 1 
        \end{array}\right]
        \left[\begin{array}{ccc}
        0  & 0  & 0\\
        1  & 0  & 0\\
        0  & 1  & 0\\
        0  & 0  & 1
        \end{array}\right]
        =
        \left[\begin{array}{ccc}
        1 & 0 & 0 \\
        1 & 0 & 0 \\
        0 & 1 & 0 \\
        0 & 1 & 0 \\
        0 & 0 & 1 \\
        0 & 0 & 1 
        \end{array}\right]
\]
which is the same design matrix that we got ``by eye'' above.

\paragraph{Delete the last column.} Some computer packages set
$\theta_3=0$ instead of $\theta_0=0$. If we set $\theta_3=0$, this is
like deleting the last column in the matrix formulation.
\[
        \left[\begin{array}{c}
        y_{11} \\ y_{12} \\ y_{21} \\ y_{22} \\ y_{31} \\ y_{32}
        \end{array}\right]
        =
        \left[\begin{array}{ccc}
        1 & 1 & 0  \\
        1 & 1 & 0  \\
        1 & 0 & 1  \\
        1 & 0 & 1  \\
        1 & 0 & 0  \\
        1 & 0 & 0  
        \end{array}\right]
        \left[\begin{array}{c}
        \theta_0 \\ \theta_1 \\ \theta_2 
        \end{array}\right]
        +
        \left[\begin{array}{c}
        \epsilon_{11} \\ \epsilon_{12} \\ \epsilon_{21} \\ 
        \epsilon_{22} \\ \epsilon_{31} \\ \epsilon_{32}
        \end{array}\right]
\]
and \eq{basic-1} becomes
\[
        y_{ij} = \theta_0 + X_{1j}\theta_1 + X_{2j}\theta_2 
          + \epsilon_{ij}
\]
We can also do this by setting up $C$ and computing
$Z=XC'(CC')^{-1}$.  We want to set $\theta_3=0$ and to define
\[
        \left[\begin{array}{c}
        \gamma_0 \\ \gamma_1 \\ \gamma_2 
        \end{array}\right]
        = 
        \left[\begin{array}{cccc}
        1 & 0 & 0 & 0 \\
        0 & 1 & 0 & 0 \\
        0 & 0 & 1 & 0
        \end{array}\right]
        \left[\begin{array}{c}
        \theta_0 \\ \theta_1 \\ \theta_2 \\ \theta_3 
        \end{array}\right]
        = C\theta
\]
It then follows that
\[
        Z = X C'(CC')^{-1} = 
        \left[\begin{array}{cccc}
        1 & 1 & 0 & 0 \\
        1 & 1 & 0 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 0 & 1 \\
        1 & 0 & 0 & 1 
        \end{array}\right]
        \left[\begin{array}{ccc}
        1  & 0  & 0\\
        0  & 1  & 0\\
        0  & 0  & 1\\
        0  & 0  & 0
        \end{array}\right]
        =
        \left[\begin{array}{ccc}
        1 & 1 & 0  \\
        1 & 1 & 0  \\
        1 & 0 & 1  \\
        1 & 0 & 1  \\
        1 & 0 & 0  \\
        1 & 0 & 0  
        \end{array}\right]
\]
as expected.  What do the parameters mean here?
\begin{eqnarray*}
        \mu_1 & = & \gamma_0 + \gamma_1 \\
        \mu_2 & = & \gamma_0 + \gamma_2 \\
        \mu_3 & = & \gamma_0.
\end{eqnarray*}
Hence, $\gamma_0 = \mu_3$ is the mean of the last cell, and $\gamma_1$ and
$\gamma_2$ are the differences between the third cell and the first
and second (respectively).

\paragraph{Grand mean plus treatment effects.} If we set the constraint
\[
        \theta_1 + \theta_2 + \theta_3 = 0
\]
then we can express $\theta_3 = -\theta_1 - \theta_2$ and \eq{basic-1}
becomes
\[
        y_{ij} = \theta_0 + X_{1j}\theta_1 + X_{2j}\theta_2 -
        X_{3j}(\theta_1 + \theta_2) + \epsilon_{ij}
\]
or equivalently
\begin{equation}
        y_{ij} = \theta_0 + (X_{1j}-X_{3j})\theta_1 + (X_{2j}-X_{3j})\theta_2 
        + \epsilon_{ij}
        \label{sums-1}
\end{equation}
and the matrix form becomes
\[
        \left[\begin{array}{c}
        y_{11} \\ y_{12} \\ y_{21} \\ y_{22} \\ y_{31} \\ y_{32}
        \end{array}\right]
        =
        \left[\begin{array}{rrr}
        1 &  1 &  0  \\
        1 &  1 &  0  \\
        1 &  0 &  1  \\
        1 &  0 &  1  \\
        1 & -1 & -1  \\
        1 & -1 & -1  
        \end{array}\right]
        \left[\begin{array}{c}
        \theta_0 \\ \theta_1 \\ \theta_2  
        \end{array}\right]
        +
        \left[\begin{array}{c}
        \epsilon_{11} \\ \epsilon_{12} \\ \epsilon_{21} \\ 
        \epsilon_{22} \\ \epsilon_{31} \\ \epsilon_{32}
        \end{array}\right]
\]

Now let's try setting up $C$ and $Z = XC'(CC')^{-1}$:
\begin{description}

\item[Step 1.] This parametrization can be thought of as doing two
things: (a) we set $\theta_0=0$; and (b) we define:
\begin{eqnarray*}
        \mu & = & (\theta_1 + \theta_2 + \theta_3)/3 \\
        \alpha_1 & = & \theta_1 - \mu ~ = ~ \phantom{\,+\,} \frac23 \theta_1 -
                        \frac13\theta_2 - \frac13\theta_3 \\
        \alpha_2 & = & \theta_2 - \mu ~ = ~ -\frac13 \theta_1 +
                        \frac23\theta_2 - \frac13\theta_3 \\
        \alpha_3 & = & \theta_3 - \mu ~ = ~ -\frac13 \theta_1 -
                        \frac13\theta_2 + \frac23\theta_3 
\end{eqnarray*}
(as a check, note that $\alpha_1 + \alpha_2 + \alpha_3=0$ as desired).

\item[Step 2.] This means that $\alpha_3 = -\alpha_1 - \alpha_2$, so
we can omit it from the reparametrization, and so $C$ is the matrix in
the transformation
\[
        \left[\begin{array}{c}
        \mu \\ \alpha_1 \\ \alpha_2 
        \end{array}\right]
        = 
        \left[\begin{array}{rrrr}
        0 &  \frac13 &  \frac13 &  \frac13 \\
        0 &  \frac23 & -\frac13 & -\frac13 \\
        0 & -\frac13 &  \frac23 & -\frac13
        \end{array}\right]
        \left[\begin{array}{c}
        \theta_0 \\ \theta_1 \\ \theta_2 \\ \theta_3 
        \end{array}\right]
        = C\theta
\]
\item[Step 3.] Therefore
\[
        Z = X C'(CC')^{-1} =
        \left[\begin{array}{cccc}
        1 & 1 & 0 & 0 \\
        1 & 1 & 0 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 0 & 1 \\
        1 & 0 & 0 & 1 
        \end{array}\right]
        \left[\begin{array}{rrr}
        0 &  0 &  0  \\
        1 &  1 &  0  \\
        1 &  0 &  1  \\
        1 & -1 & -1  
        \end{array}\right]
        =
        \left[\begin{array}{rrr}
        1 &  1 &  0  \\
        1 &  1 &  0  \\
        1 &  0 &  1  \\
        1 &  0 &  1  \\
        1 & -1 & -1  \\
        1 & -1 & -1  
        \end{array}\right]
\]
so the new model is
\[
        y = 
        \left[\begin{array}{rrr}
        1 &  1 &  0  \\
        1 &  1 &  0  \\
        1 &  0 &  1  \\
        1 &  0 &  1  \\
        1 & -1 & -1  \\
        1 & -1 & -1  
        \end{array}\right]
        \left[\begin{array}{c}
        \mu \\ \alpha_1 \\ \alpha_2  
        \end{array}\right]
        + \epsilon
\]

\item[Step 4.] Now if we define 
\[
Z_1 = X_1 - X_3 =
        \left\{\begin{array}{rr}1 & \mbox{at level $a$} \\
                              0 & \mbox{at level $b$} \\
                             -1 & \mbox{at level $c$}
                                \end{array}\right.
~~~~~~~~~~~~
Z_2 = X_2 - X_3 =
        \left\{\begin{array}{rr}0 & \mbox{at level $a$} \\
                              1 & \mbox{at level $b$} \\
                             -1 & \mbox{at level $c$} 
                                \end{array}\right.
\]
we may express the model as
\[
        y_{ij} = \mu + Z_{1j}\alpha_1 + Z_{2j}\alpha_2 
        + \epsilon_{ij}
        ~ = ~
        \mu + (X_1 - X_3)\alpha_1 + (X_2 - X_3)\alpha_2 
        + \epsilon_{ij}
\]
In this model, the cell means are
\begin{eqnarray*}
        \mu_1 & = & \mu + \alpha_1 \\
        \mu_2 & = & \mu + \alpha_2 \\
        \mu_3 & = & \mu + \alpha_3 
\end{eqnarray*}
so that the $\alpha$'s really are offsets from the grand mean $\mu$ to
each cell mean.

\end{description}

\paragraph{SPLUS's default ``Helmert contrasts''.}  The default
parametrization for ANOVA models in SPLUS is in terms of the ``Helmert
contrasts.''  It is easiest to explain how to set this model up
directly in terms of $C$ and $Z = XC'(CC')^{-1}$:

\begin{description}

\item[Step 1.] Like the ``grand mean plus treatment effects''
parametrization we do two things: (a) we set $\theta_0=0$; and (b) we
define:
\begin{eqnarray*}
        \mu & = & (\theta_1 + \theta_2 + \theta_3)/3 \\
        \beta_1 & = & \{-\theta_1 + \theta_2\}/2 \\
        \beta_2 & = & \{-(\theta_1 +\theta_2)/2 + \theta_3\}/3 \\
        \beta_3 & = & \{-(\theta_1 + \theta_2 + \theta_3)/3 + \theta_4\}/4
\end{eqnarray*}
{\em (Here, $\theta_4\equiv0$ [since there is no fourth category] and
is just included to show you the pattern, should you ever have to
construct Helmert contrasts for more than three categories).}

\item[Step 2.] Again, $\beta_3$ is expressible in terms of the other
parameters ($\beta_3 = 3\mu$), so we omit it from the
reparametrization, and $C$ is the matrix in the transformation
\[
        \left[\begin{array}{c}
        \mu \\ \alpha_1 \\ \alpha_2 
        \end{array}\right]
        = 
        \left[\begin{array}{rrrr}
        0 &  \frac13 &  \frac13 &  \frac13 \\
        0 & -\frac12 &  \frac12 & 0 \\
        0 & -\frac16 & -\frac16 &  \frac13
        \end{array}\right]
        \left[\begin{array}{c}
        \theta_0 \\ \theta_1 \\ \theta_2 \\ \theta_3 
        \end{array}\right]
        = C\theta
\]
\item[Step 3.] Therefore
\[
        Z = X C'(CC')^{-1} =
        \left[\begin{array}{cccc}
        1 & 1 & 0 & 0 \\
        1 & 1 & 0 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 1 & 0 \\
        1 & 0 & 0 & 1 \\
        1 & 0 & 0 & 1 
        \end{array}\right]
        \left[\begin{array}{rrr}
        0 &  0 &  0  \\
        1 & -1 & -1  \\
        1 &  1 & -1  \\
        1 &  0 &  2  
        \end{array}\right]
        =
        \left[\begin{array}{rrr}
        1 & -1 & -1  \\
        1 & -1 & -1  \\
        1 &  1 & -1  \\
        1 &  1 & -1  \\
        1 &  0 &  2  \\
        1 &  0 &  2  
        \end{array}\right]
\]
so the new model is 
\[
        y = 
        \left[\begin{array}{rrr}
        1 & -1 & -1  \\
        1 & -1 & -1  \\
        1 &  1 & -1  \\
        1 &  1 & -1  \\
        1 &  0 &  2  \\
        1 &  0 &  2  
        \end{array}\right]
        \left[\begin{array}{c}
        \mu \\ \beta_1 \\ \beta_2  
        \end{array}\right]
        + \epsilon
\]

\item[Step 4.] Now if we define 
\[
Z_1 = X_2 - X_1 =
        \left\{\begin{array}{rr}-1 & \mbox{at level $a$} \\
                              1 & \mbox{at level $b$} \\
                              0 & \mbox{at level $c$}
                                \end{array}\right.
~~~~~~~~~~~~
Z_2 = 2X_3 - X_2 - X_1 =
        \left\{\begin{array}{rr}-1 & \mbox{at level $a$} \\
                             -1 & \mbox{at level $b$} \\
                              2 & \mbox{at level $c$} 
                                \end{array}\right.
\]
we may express the model as
\[
        y_{ij} = \mu + Z_{1j}\beta_1 + Z_{2j}\beta_2 
        + \epsilon_{ij}
        ~ = ~
        \mu + (X_2 - X_1)\beta_1 + (2X_3 - X_2 - X_1)\beta_2 
        + \epsilon_{ij}
\]
In this model, the cell means are
\begin{eqnarray*}
        \mu_1 & = & \mu - \beta_1 - \beta_2 \\
        \mu_2 & = & \mu + \beta_1 - \beta_2 \\
        \mu_3 & = & \mu + 2\beta_2 
\end{eqnarray*}
Note that it is easy to read off the ``treatment effects'' also:
\begin{eqnarray*}
        \alpha_1 & = & - \beta_1 - \beta_2 \\
        \alpha_2 & = & \phantom{\,+} \beta_1 - \beta_2 \\
        \alpha_3 & = & \,\,2\beta_2 
\end{eqnarray*}


\end{description}

\end{document}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% \section{Two-Way Layouts} 
% 
% Some terminology:
% \begin{itemize}
% 
% \item
% The data for an ANOVA model involving a single factor is sometimes
% called a {\em ``one-way layout''}, since a table of the data would
% usually be laid out one way along the factor as a row or column.
% 
% \item
% When there are two factors, the data is usually laid out as a two
% dimensional table with the levels of the first factor in rows (say)
% and levels of the second factor in columns; this is called a {\em
% ``two-way layout''}, and the corresponding model is called a {\em
% two-way ANOVA model}.
% 
% \end{itemize}
% 
% 
% 
% 
% \section{\boldmath{$2^k$} Layouts}
% 
% 