---
title: Neural Networks
author: 36-462/662, Spring 2022
date: 5 April 2022 (Lecture 21)
bibliography: locusts.bib
header-includes:
  - \DeclareMathOperator*{\argmin}{argmin}
output:
  pdf_document:
    toc: true
---

\newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)}
\newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}}
\newcommand{\Loss}{\ell}
\newcommand{\EmpRisk}{\hat{R}}
\newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}

# From Logistic Regression to Multi-Layer Neural Networks

Let's look again at the probability which is output by logistic regression
(rather than the class label).  It's
\begin{equation}
\Prob{Y=1|\vec{X}=\vec{x}} = \frac{e^{b+\vec{x}\cdot\vec{w}}}{1+e^{b+\vec{x}\cdot\vec{w}}}
\label{eqn:logistic-response}
\end{equation}
This takes an input vector $\vec{x} = (x_1, x_2, \ldots x_p)$ and calculates a
weighted sum of its components, $\vec{x} \cdot \vec{w} = \sum_{j=1}^{p}{x_j
  w_j}$.  It then adds on a constant.  This can give us a number, call it
$\eta$, anywhere between $-\infty$ and $+\infty$.  We then calculate
$e^{\eta}/(1+e^{\eta})$.  This squashes the number back into the range $[0,1]$.
To speak like an engineer, the output $p$ is very nearly linear in $\eta$ when
$\eta \approx 0$, but when $\eta$ is very large ($\rightarrow \infty$) or very
small ($\rightarrow -\infty$), the output **saturates**.  This kind of
behavior is typical of lots of nonlinear devices, and in particular it serves
as a crude caricature of how nerve cells or **neurons** behave.  This led to
the idea that if we can compute simple things, like classification, with one
device of this sort, maybe we can compute more complicated functions with many
of them wired together.

Formally, a **feed-forward artificial neural network** is defined by having
**layers** of such devices, where the output of one layer provides the input to
the next.  The convention is to count the input variables as the first layer,
so logistic regression is an example of a two-layer neural network.  If we add
an **intermediate** or **hidden** layer with $m$ nodes between the inputs
and the outputs, one has a model like so:
\begin{eqnarray}
t_k & = & f_h(\alpha^{(1)}_k + \sum_{l=1}^{p}{x_l w^{(1)}_{lk}}) \\
& = & f_h(\eta_k(\vec{x}))\\
s_j & = & f_o(\alpha^{(2)}_j + \sum_{k=1}^{r}{t_k w^{(2)}_{kj}})\\
& = & f_o(\xi_j(\vec{x}))
\end{eqnarray}
Here we have an input vector $\vec{x}$, of dimension $p$, and an output vector
$\vec{s}$, of dimension $q$.  $t_k$ is the output of the $k^{\mathrm{th}}$
hidden node (there are $m$ of them).  $f_h$ is the **activation**, **response** or **squashing** function of the hidden nodes, and $f_o$ is the
activation function of the output nodes.  The weights $w^{(1)}_{lk}$ and $w^{(2)}_{kj}$
control how strongly the output of $k$ is influenced by $j$.  Similarly the
biases $\alpha^{(1)}$ and $\alpha^{(2)}$ set base-line levels of activation for each
node[^notationvariants].  I will write the over-all output of the neural
network on inputs $\vec{x}$ as $\phi(\vec{x}; \mathbf{\alpha}, \mathbf{w})$.
It is, I hope, clear how we could extend this notation to multiple intermediate
or hidden layers, which will become important later.


[^notationvariants]: It's common to introduce an extra input variable, call it $x_0$ or $t_0$, which is always equal to 1.  Then we can replace the off-sets $\alpha^{(1)}_k$ and
$\alpha^{(2)}_j$ with weights $w^{(1)}_{0l}$ and $w^{(2)}_{0j}$.  Also, some people like to allow
layer-skipping connections, directly from inputs to outputs; this raises no
point of principle but complicates the notation a bit.  Finally, some people
like to number the units consecutively across all the layers, so that one
just needs a single weight matrix $w$, rather than two.  Again, this doesn't
raise any point of principle.

Using a three-layer neural network thus deciding on the number $m$ of hidden
units, what the activation functions should be, and on what the weights should
be.

Take $m$ as given for now; we'll come back to that.

The activation functions are almost always chosen to be monotonic, and if we
want to be able to estimate efficiently they should also be smooth.  (We'll
come back to why in a little bit.)  Otherwise, appropriate choices depends on
what we want to do.  Common activation functions are linear (the identity
function),
\begin{equation}
f(\eta) = \eta ~,
\end{equation}
"rectified linear" (a.k.a. "ReLU" for "rectified linear unit"),
\begin{equation}
f(\eta) = \eta \Indicator{\eta > 0}
\end{equation}
logistic
\begin{equation}
f(\eta) = e^{\eta}/(1+e^{\eta} ~,
\end{equation}
and the hyperbolic tangent
\begin{equation}
f(\eta) = \tanh{\eta} = \frac{e^{\eta} - e^{-\eta}}{e^{\eta} + e^{-\eta}}
\end{equation}
Most often, $f_h$ is the
hyperbolic tangent, and $f_o$ is linear for regression problems, or logistic
for classification problems[^softmax].

[^softmax]: Another way to approach classification is to have one output node for each class, with a linear activity function, and then take the probability of class $j$ to be $e^{s_j}/\sum_{j=1}^{q}{e^{s_j}}$; this ensures that probabilities sum to 1.

So: fix the number of neurons, fix the activation functions, and "all" that's
left to find are the weights.  This is called **training**.

# Gradient Descent and Backpropagation

People usually train neural networks by gradient descent.  Remember that
we have training data $(\vec{x}_1, y_1), \ldots (\vec{x}_n, y_n)$, and a
loss function $\Loss$.  If we set all the weights and biases, the empirical
risk will be
\begin{equation}
\EmpRisk(\mathbf{\alpha}, \mathbf{w}) = \frac{1}{n}\sum_{i=1}^{n}{\Loss(y_i, \phi(\vec{x}_i; \mathbf{\alpha}, \mathbf{w}))}
\end{equation}
(The empirical risk is implicitly a function of the training data, but I am
suppressing that in the notation for now.)

If we want to optimize this, we're going to need to take derivatives
with respect to the parameters.

## Output-layer derivatives

Taking the derivative with respect to parameters for the output layer is
straightforward, if long-winded, using the chain rule:
\begin{equation}
\frac{\partial \EmpRisk}{\partial w^{(2)}_{jk}} = \frac{1}{n}\sum_{i=1}^{n}{\left. \frac{\partial \Loss(y_i, a)}{\partial a_j}\right|_{a=\phi(\vec{x}_i; \mathbf{\alpha}, \mathbf{w})} \left. \frac{d f_o}{d u}\right|_{u=\xi_j(\vec{x}_i)} t_k(\vec{x}_i)}
\end{equation}

This is a mouthful, so let's think it through for a simple case, specifically
regression with a squared-error loss function.  Because we're predicting a
single real number, we have _one_ output node, hence $q=1$.  The derivative
$\frac{\partial \Loss(y_i, a)}{\partial a} = 2(y_i - a)$, a.k.a. (twice)
the residual.  So in the regression case, we'd get
\begin{equation}
\frac{\partial \EmpRisk}{\partial w^{(2)}_{j1}} = \frac{1}{n}\sum_{i=1}^{n}{\text{residual}_i \left. \frac{d f_o}{d u}\right|_{u=\xi(\vec{x}_i)} t_j(\vec{x}_i)}
\end{equation}
The first factor in the sum is "how does the loss change (on this data point)
if the output is increased slightly?"  The second factor in the sum is "how
much does the output change if the total input to that node is increased
slightly?"[^derivativeofactivationfunction]  The third factor is "how much does the total input to the node
change if the weight we're interested in is increased slightly?"

[^derivativeofactivationfunction]: Also, notice that this is the only place where the shape of the activation function appears explicitly.  If the activation function was linear, this derivative would always be 1.  If the activation function is "rectified linear", the derivative is always 0 (below the threshold) or 1 (above it); this simplicity is one of the reasons ReLU neural networks are popular.  If we use a logistic or hyperbolic-tangent activation function we need to do more calculation, but the derivative of $\tanh{u}$ is (exercise!) $\left(\frac{2}{e^u + e^{-u}}\right)^2$ which is easy enough.  Further exercise: find the derivative of the logistic activation function.


If we are using a different loss function, all that changes is that the first
factor is no longer (twice) the residual, but some other slope saying "how much
would the loss on this data point increase if the output of this final-layer
neuron increased a little?"  You can, if you like, think of such a slope as a
sort of generalized residual.  If we've got multiple output-layer neurons,
we need to take the derivatives for each of them, but that's a calculation
we can do "in parallel" for all of them.

(I will leave it as a character-building exercise to take the derivatives with
respect to the biases $\alpha^{(2)}_k$.)

## Hidden-layer derivatives

Now suppose we want to take the derivative with respect to one of the weights
going into the hidden layer, $w^{(1)}_{lk}$.  We use the chain rule again, and
we recognize that changing $w^{(1)}_{lk}$ is only going to change the loss if
doing so changes the activation of an output node, which is only going to
happen if doing so changes the total input to an output node.  So we
get, from the chain rule,
\begin{eqnarray}
\lefteqn{\frac{\partial \EmpRisk}{\partial w^{(1)}_{lk}}} & & \\
\nonumber & = & \frac{1}{n}\sum_{i=1}^{n}{\sum_{j=1}^{q}{\left. \frac{\partial \Loss(y_i, a)}{\partial a_j}\right|_{a=\phi(\vec{x}_i; \mathbf{\alpha}, \mathbf{w})} \left. \frac{d f_o}{d u}\right|_{u=\xi_j(\vec{x}_i)}} w^{(2)}_{jk} \left. \frac{d f_h}{dv}\right|_{v=\eta_k(\vec{x}_i)} x_{il}}
\end{eqnarray}
Again, this is a mouthful, but again, we can make some sense of it by taking
it piece by piece.

The first factor in each term being averaged is the generalized residual for
that data point.  The second factor is how much the output of each final-layer
unit changes as we change its total input.  The third factor is how much the
total input to final-layer unit $j$ changes in response to a small change in
the output of hidden-layer unit $k$.  We need to sum these three factors,
multiplied togehter, over all final-layer units, because we're contemplating a
change to how the hidden-layer unit $k$ works, which could affect any or all of
the final-layer units.  (If we're doing regression or binary classification we
only have one final-layer unit, so that sum over $j$ would go away.)  We then
weight this by how much the output of unit $k$ changes in response to a small
change in its total input, and then by how much a small change in the weight
unit $k$ gives to feature $l$ will change the total input.

You can imagine extending this if there were multiple hidden or intermediate
layers.  Doing so will illuminate why people who are into neural networks are
also into automatic differentiation.

## The backpropagation trick

Something to notice about what we've just done is that a _lot_ of the work for
the hidden-layer units is also needed for the final-layer units.

Let's say that for each final-layer unit $j$, we calculate
\begin{equation}
b_{ij} = \left. \frac{\partial \Loss(y_i, a)}{\partial a_j}\right|_{a=\phi(\vec{x}_i; \mathbf{\alpha}, \mathbf{w})} \left. \frac{d f_o}{d u}\right|_{u=\xi_j(\vec{x}_i)} \label{eqn:back-prop-signal-final}
\end{equation}
which is the generalized residual for that unit, times the slope of
the unit's activation function: "how much would increasing the output of $j$
change the loss, _times_ how rapidly does the output change when we change
the total input?"

Similarly, for each hidden-layer unit $k$, we calculate
\begin{equation}
b_{ik} = \sum_{j=1}^{q}{b_{ij} w_{jk} \left. \frac{d f_h}{dv}\right|_{v=\xi_k(\vec{x}_i)}} \label{eqn:back-prop-signal-hidden}
\end{equation}
That is, we sum the $b_{ij}$ weighted by $w_{jk}$, and we multiply everything by the derivative of $k$'s output with respect to its total input.

Now 
\begin{equation}
\frac{\partial \EmpRisk}{\partial w^{(2)}_{jk}} = \frac{1}{n}\sum_{i=1}^{n}{b_{ij} t_j(\vec{x}_i)} \label{eqn:gradient-from-back-prop-final}
\end{equation}
but
\begin{equation}
\frac{\partial \EmpRisk}{\partial w^{(1)}_{lj}} = \frac{1}{n}\sum_{i=1}^{n}{b_{ik} x_{il}} \label{eqn:gradient-from-back-prop-hidden}
\end{equation}

We can extend this to even more layers, if we need to, by defining $b_{il}$
similarly: the sum of the $b_{ik}$ over the neurons in the next layer, times he
weights, times the slope of the activation function.

This idea was introduced by @Rumehlart-Hinton-Williams-introduce-backprop, under the name of **back-propagating
error** or **back-propagation**, and what I have called the $b_{ij}$,
$b_{ik}$, $b_{il}$ are the **back-propagation signals**.  These
provide an extremely handy way of calculating the derivatives we need
for optimizing the weights[^backprophistory].

[^backprophistory]: While @Rumehlart-Hinton-Williams-introduce-backprop made the technique a permanent part of the neural-network literature, and introduced the handy name "back-propagation", there is a complicated history of independent re-inventions, of greater or lesser generality, going back to the 1960s at least.  (Since it's just an application of the chain rule, in principle it could have been discovered at any time since the early 18th century.)  The reference works given under "Further Reading" below discuss some of this history.

### Gradient descent and stochastic gradient descent

Here's a classic way of doing gradient descent for neural networks:

- Start with some initial weights (and biases) for all the units.
- Set $t=1$
- Until the weights stabilize, or $t = t_{max}$:
    + Cycle through the data points from $1$ to $n$
        * For each data point $i$, compute the back-propagation signals $b_{ij}$ and then $b_{ik}$ using the current weights
		* Adjust the weights: $w_{jk} \rightarrow w_{jk} - \gamma_t b_{ij} t_j(\vec{x}_i)$, $w_{lk} \rightarrow w_{lk} - \gamma_t b_{ik} x_{il}$, where the **gains** $\gamma_t$ gradually decrease towards zero
		* $t \rightarrow t_+1$
		
You can, of course, imagine all kinds of variations here.  For instance, maybe
we can save a lot of time by calculating the back-propagation signals once on
each pass through the data, rather than adjusting after each data point;
maybe we change the order in which we visit data points on each pass.

One particularly important variation is that we might not visit_every_ data
point on each cycle.  In particular, we might randomly sample a smaller number
of data points $r$, perhaps even just $r=1$.  As we saw when we looked at such
stochastic gradient descent in general, _in expectation_ this gives the correct
derivatives, and that's still true when we use back-propagation to calculate
the derivatives.  This can be _extremely_ useful when the data size is very
large.

This combination of back-propagation to efficiently calculate derivatives, and
gradient descent or stochastic gradient descent, is not the only possible way
to train neural networks, but it's overwhelmingly the most popular.

# How many layers? How many neurons per layer?

## Two-layer neural networks, a.k.a. "perceptrons"

Two-layer neural networks, also called "perceptrons" after @Rosenblatt-perceptron, are pretty straightforward:
\begin{equation}
s_j = f_o(\alpha_j + \sum_{l=1}^{p}{x_{l} w_{jl}})
\end{equation}
We take a linear combination of the input features and apply a single
nonlinearity to them (e.g., a logistic transformation).  These are what
econometricians would call **single-index models**.  (What's that single
index?)  Because $f_o$ is monotonic, all the classifications we can realize by
thresholding the output are ones we could achieve _without_ the transformation,
by applying a different threshold to the linear form $\alpha_j +
\sum_{l=1}^{p}{x_{l} w_{jl}}$.

So: two-layer neural networks are just a slight variant on linear models.  In
particular, as classifiers they really are just linear classifiers.

## Three-layer neural networks, a.k.a. multi-layer perceptrons

Three-layer neural networks are vastly more powerful than two-layer neural
networks.  In particular, with the right choice of smooth, nonlinear
activation functions (logits and hyperbolic tangent very much included),
we have (roughly) the following result:

> Let $g: \mathbb{R}^p \mapsto \mathbb{R}^q$ be any smooth function, and pick any limited domain $D \in \mathbb{R}^p$.  Then, by taking $m$ large enough, we can always find a three-layer network with $m$ nodes and weights $\mathbf{w}$ so that $\max_{\vec{x} \in D}{\|g(\vec{x}) - \phi(\vec{x}; \mathbf{w})\|} \leq \epsilon$.

That is, by throwing enough hidden units at the problem, we can approximate an
arbitrary (smooth) function arbitrarily closely.  We say that three-layer
neural networks are **universal approximators**.

Now, many other combinations of basic functions can also be universal
approximators, in this sense.  The most obvious are polynomials (a result known
as **Weierstrass's theorem**), but one can also use combinations of sines and
cosines, and many other function families; the general result in mathematical
analysis which covers this is what's called the **Stone-Weierstrass* theorem.
(Indeed, most proofs of the result I just gave use the Stone-Weierstrass
theorem.)  But it's still remarkable that adding just one layer increases the
power of the models so remarkably, from "basically linear" to "basically
anything".

### Deriving new features, and cautions about interpretation

The final layer of the neural network is doing a (basically) linear operation
on the output of the hidden layer.  (Or, with even more layers, on the output
of the next-to-final layer.)  So it is very tempting to think that what's
happening is that the hidden layer is computing new features from the input
variables, and then the final layer is applying a linear method to those new
features.  Thus training the neural network is in some sense discovering new
features, or learning a new representation of the data.

This is, I think, basically right, but it requires the caution that it's
very hard to interpret those new features.

Specifically, one is tempted to try to give some human, understandable story
about what each hidden-layer unit is interpreting.  One common procedure for
coming up with such a story about hidden-layer unit $k$ is to take the trained
network, and then look at the input vectors $\vec{x}_i$ which maximize $t_k$.
Then one tries to figure out what those input vectors have in common, and so
what unit $k$ is representing.

The difficulty here is that unit $k$ doesn't really do _anything_ on its own,
but only in concert with all the other hidden-layer units.  The hidden layer as
a whole provides a representation of the data, but it's generally a
**distributed representation**, not (necessarily) localized to particular
units.  Neuroscientists studying biological networks long ago coined the term
"grandmother cell fallacy" to describe the difficulty: The fact that you can
recognize your grandmother's face (and distinguish it from other people's
faces, from her hands, from non-faces, etc.) means that the biological neural
network in your brain must, somehow, represent your grandmother's face.  It
does not follow that there is one particular neuron which is activated when you
see her face!

So, as a point of theory, we shouldn't _expect_ individual hidden-layer units
to represent humanly-meaningful features, even if the hidden layer acts as a
distributed representation.  Concretely, @intriguing-properties did the
experiment of creating random weights $Z_k$ and looking at the inputs which
maximizes $\sum_{k=1}^{m}{Z_k t_k(\vec{x})}$ in trained neural networks that
worked well at image classification[^multilayer].  The images that maximized
the output of random combinations of hidden-layer units were just as coherent,
and just as easy to tell stories about and interpret, as those which maximized
the activation of individual hidden-layer units[^humanrademacher].  Again, this is compatible
with the hidden layer _as a whole_ being a distributed representation of the
data, in terms of new features.

[^multilayer]: Actually, the networks they looked at were "deep" networks with many more than three layers, but the idea is the same.

[^humanrademacher]: @human-rademacher-complexity, in a series of experiments which deserve to be much better known than they are, showed that humans --- or at least undergrads at UW-Madison --- have a pretty impressive ability to come up with stories that let them interpret, and memorize, quite long collections of literally-random dictionary words.  It would be very interesting to know how big a collection of random images from the data sets used to train these neural networks could be "interpreted", and memorized, by graduate students and professors of machine learning.


### Auto-encoders, dimension reduction, embeddings

The universal approximations result say that with _large enough_ hidden layers,
we can come arbitrarily close to any function we like.  This often leads to
situations where $m > p$, so we expand out into a higher-dimensional space.
(This should remind you of the kernel trick.)  There are particularly
interesting situations, however, where $m < p$, so we try to **encode** or
**compress** the relevant aspects of the input into a smaller number of
dimensions.  This doesn't always work --- there isn't a guarantee that a good
compression exists --- but when we can pull it off we've done dimension
reduction.

An extremely interesting example of the bottleneck situation is when the output
dimension and the input dimensions are equal, $q=p$, and we use a loss function
which simply rewards reproducing the input, _but_ the hidden layer has fewer
units, $m < p$.  A network which can do this is called an **auto-encoder**.
This will be possible when the original high-dimensional input is actually
highly redundant, and, to a good approximation, has a much smaller number of
dimensions, _and_ we can find a way of recovering the input from that
compressed representation.  This should remind you of PCA; you can think of
auto-encoders as being like a nonlinear sort of PCA, or PCA as being a linear
auto-encoder.

When we _can_ make $m \ll p$ and still get good results, we sometimes refer to
the hidden layer as a **bottleneck** which the information needs to pass
through.  Exactly what the connections are here to the "information bottleneck
method" introduced by @Tishby-Pereira-Bialek-bottleneck is an active subject of
investigation, but many researchers strongly suspect that this is a big part of
what's going on in neural networks.

If we have an input $x$ which the low-dimensional hidden layer transforms into
a point $\vec{t}(x) \equiv (t_1(x), t_2(x), \ldots t_m(x)) \in \mathbb{R}^m$,
we sometimes refer to $\vec{t}(x)$ as an **embedding** of $x$ into the vector
space $\mathbb{R}^m$.  Once we've represented the input, whatever it might be,
as a point in a vector space, we can do all kinds of things to it, like do
similarity search, or analogy-completion tasks (cf. @Kearns-Roth-ethical,
pp. 57--63).  Again, you can think of this as one way of generalizing
what we did with PCA.

An outstanding example of successfully-applied neural network embeddings is
provided by word embeddings, beginning with @word2vec.  The basic idea here is
to try to predict the occurrence of a word in text as a function of the other
words around it, with each word being represented by a not-too-high-dimensional
vector.  The original paper on this `word2vec` technique was sufficiently
opaque to lead to a series of follow-up papers explaining what, exactly, was
going on (like @Goldberg-Levy-word2vec-explained), but it also worked really
well on lots of natural-language-processing tasks.  In particular, it worked
much better than previous methods of embedding words in vector spaces based on
PCA [@dont-count-predict].  Since @word2vec appeared, neural word embedding
models have gotten much more elaborate, and there has been a flurry of effort
to devise similar sorts of embedding methods for other kinds of data, e.g.,
nodes in graphs.


### Random features

Remember that when we looked at [random features](https://www.stat.cmu.edu/~cshalizi/dm/22/lectures/15/lecture-15.pdf), we looked at models of the
form
\begin{equation}
s(\vec{x}) = \sum_{k=1}^{m}{w_k f(\vec{x}; \omega_k)}
\end{equation}
where $f$ was a _fixed_ nonlinear function, and the parameters $\omega_k$ were
drawn randomly from some distribution $\rho$, leaving only the weights $w_k$ to
be adjusted to minimize the risk.  In particular, we got a lot of good results
with random Fourier features, so $f(\vec{x}; \omega) = \cos{(\vec{x} \cdot
\nu + \delta)}$, breaking the $(p+1)$-dimensional $\omega$ into a
$p$-dimensional "wave vector" $\nu$ and a scalar "phase" $\delta$.  However,
there was nothing magical (as opposed to convenient) to using random
Fourier features, as opposed to, say, random hyperbolic tangent features.

We introduced random feature models as a way of efficiently approximating
kernel machines when the number of training data points grew large --- instead
of dealing with an $n\times n$ kernel matrix, we could deal with just an
$m\times m$ covariance matrix of the random features, and get nearly the same
results.  We can see now that _another_ way of understanding random feature
models is as three-layer neural networks, where the weights connecting the
input features to the hidden layer are randomly generated independently of the
data, _and then fixed_ during training.  Instead, only the weights
connecting the hidden layer to the output layer are optimized, eliminating
the need for back-propagation.


## Deep neural networks and deep learning

A "deep" neural network, as opposed to a shallow one, is simply one with more
than three layers --- the more layers it has, the deeper it is.  "Deep
learning" is simply fitting a deep neural network, which is, overwhelmingly,
done by the combination of  back-propagation and stochastic gradient descent
sketched above.  (See more below.)

Over the last ten years or so, deep neural networks have proved _extremely_
effective at a lot of tasks relating to image classification, speech
recognition, and natural language processing.  This is actually puzzling,
because nobody has managed to get equivalent performance out of shallow-but-wide
three-layer networks, even though we know, mathematically, that deep
networks have no more expressive power than shallow ones.  (Once you're
already a universal approximator, there's not really much room to expand
your scope.)  This poses a lot of puzzles for statistical learning.

### Not just back-prop and SGD...

I said a moment ago that deep neural network training is _mostly_ just
back-propagation and stochastic gradient descent.  The "mostly" conceals a
lot of bells and whistles which have been added to that basic contraption,
either to speed up training or to improve the performance of the
learned machine.  We could easily spend the better part of a course going over
all these modifications.  It is not at all clear _which_ of these tweaks
actually improve training, how much they improve training, or _why_ they improve
training.  While there are people actively working on all of those problems,
I think it's fair to say that they have not achieved the level of clarity
and consensus that it'd be worth discussing the details here.  But if you
are going to work with these models on an industrial scale, then I am afraid
you are going to have to plunge into those details --- but they also change
very rapidly, so, again, I am not going to go over ideas which will
soon be obsolete.

## Choice of number of layers and number of units per layer

These are different model classes; use cross-validation.

(Or any other good way of estimating generalization error.)

## Beyond feed-forward networks

The fact that I called the models I've described "feed-forward" should suggest
that there is a "feed-back" alternative.  There are many.  The most common are
**recurrent** neural networks, where the inputs to units in early layers
include the outputs of units in later layers.  Feed-forward networks are just
ways of specifying functions, mappings from $\mathbb{R}^p$ to $\mathbb{R}^q$.
Recurrent networks aren't functions, or, if you like, aren't always the _same_
function, because the history of previous inputs matters.  They describe
dynamical systems or automata[^transducers], rather than just functions.
Training them is harder; I defer to references under "Further Reading".

[^transducers]: Specifically, what automata-theorists call **transducers**, which map sequences of inputs to sequences of outputs.


# Some history and some puzzles

Artificial neural network models, almost as we know them, originated with
@McCulloch-Pitts-immanent-calculus --- that is, with biologists and
psychologists using a deliberately-simplified model of how biological neurons
interact with each other to show that processes in the brain could implement
abstract logic[^protoconnectionism].

[^protoconnectionism]: If you look at some of the classics of late-19th century psychology, like @James-principles-of-psych, you will see lots of diagrams which _look_ an awful lot like pictures of neural networks.  These were speculations about some sort of "activity" or "energy" or "current" flowing through distinct regions of the brain, accompanying, or implementing, processes of thought.  It was only around 1900 that the great anatomist Santiago Ramon y Cajal proved that the nervous system is divided into distinct cells, which he named "neurones", rather than being (as some had held) a single continuous mass.  The idea that the nervous system  works by neurons sending _something_ from cell to cell, at junctions called "synapses", and that synapes work in one direction only, was established by the great physiologist Charles Sherrington, who at least sometimes referred to the nervous system as a "network".  (@Sherrington-integrative-action is still a remarkably insightful work.)  It wasn't until the 1920s that it became clear that neurons sent electrical impulses _to_ their synapses, but the transmission at the synapse is chemical.  It then became clear that a neuron's electrical "spike" was only triggered if the total stimulation it received crossed some threshold.  This is what @McCulloch-Pitts-immanent-calculus modeled in a deliberately simplified way, and what turned into the activiation function.

The next turn towards what we call neural networks came with
@Rosenblatt-perceptron, whose perceptrons were, in our terms, two-layer neural
networks used as linear classifiers for various pattern-recognition problems[^hebbetc].
This was greeted by the press with stories about the arrival of "thinking
machines", worries about what people would do when jobs were automated away,
domination by artificial intelligence, etc.

[^hebbetc]: I am simplifying, and in particular glossing over the extremely important work done by @Hebb-organization, @Hayek-sensory-order and the now-neglected but actually very-influential @Ashby-design.

There was _also_ serious scientific research into the possibilities and
limitations of these models.  In particular, @Minsky-Papert-perceptrons,
recognizing that perceptrons could only do linear classification, established
conclusive that there were lots of pattern-recognition tasks they just could
not do, no matter how much data they got or how they were trained.  This more
or less killed the field for almost two decades.

The revival of interest in neural networks, at the time also called
"connectionism", came mostly from psychology.  The two important ideas were the
recognition that adding one or more intermediate layers could drastically
expand the expressive power of these models, _and_ that the intermediate layers
could act as distributed representations.  In the late 1980s and the early
1990s, there was a _lot_ of interest, both scientific and popular, in the
genuinely-impressive results which people got on various artificial problems,
especially when using back-propagation
[@Rumehlart-Hinton-Williams-introduce-backprop].  There were again all kinds of
stories about thinking machines, "naturally intelligent
systems"[^caudillbutler], technological unemployment, artificial intelligence,
etc.  Interest in neural networks actually played key role in the development
of "statistical learning" as a field (e.g.,
@Anthony-Bartlett-neural-network-learning), which is why one of our leading
conferences is called "Advanced in Neural Information Processing Systems".

[^caudillbutler]: The title of one of the best of the semi-popular books at the time, @Caudill-Butler-naturally, which I still actually recommend.

Neural networks nonetheless faded from popularity for about two decades between
the mid-1990s and the early-2010s, for two reasons.

1. _As predictive models_, it turned out that people were able to get
much higher performance using _different_ kinds of models.  In particular,
kernel and support vector machines were, for a long time, the state-of-the-art,
random forests had (and have) their partisans, etc.
2. Even when neural networks could generalize in the statistical sense, i.e., predict well on new data points drawn from the same distribution, they did not find it easy to _generalize_ in ways that humans find easy.  If instead of using an $8 \times 8$ board to play checkers one introduces a $9\times 9$ board, a human player will be a bit hesitant, but not totally at a loss of what to do; a neural network would need to be completely retrained.  More generally, neural
networks have found it _extremely_ hard to incorporate abstract concepts, and general rules using those abstractions.  It's clear that abstractions and rules _can_ be implemented using neural networks, but it wasn't clear how to _learn_
them, and back-propagation, starting from pretty un-structured starting points, seemed to be very bad at doing so.  (@Marcus-algebraic-mind remains a very good statement of this line of criticism.)

The second issue is a real one for cognitive science, psychological
modeling and artificial intelligence, but less important for statistical
learning.  Nonetheless, the combination of the two put neural networks into
eclipse, again[^selfcentered].

[^selfcentered]: When I started teaching the predecessor to this course in 2006 and included neural networks, the feedback was that I was wasting the students' time with an obsolete technique, and by 2009 I'd dropped the topic.

All of this changed around 2012--2013, when people began publishing extremely
impressive _statistical_ results using deep neural networks on image
classification and natural-language-processing tasks.  There were a couple of
reasons why the new models worked better than their predecessors of 15--20 years
before.

1. New architectures.  In particular, for image-processing tasks, it turned out to be extremely important to use "convolutional" neural networks[^cnn], rather than unstructured ones.
2. Improved training.  In particular, the bells and whistles added on to
back-propagation and stochastic gradient descent that I alluded to.
3. Lots of data.  Truly startingly large data sets are now available, and can be
used in the training process, at least if you have a _lot_ of computing power.

[^cnn]: Mathematically, the convolution of two functions $f$ and $g$ on the same domain is another function $f* g$, where $(f*g)(r) = \int{f(u) g(r-u) du}$.  On a discrete domain, we'd have instead $(f*g)(r) = \sum_{u}{f(u) g(r-u)}$.  In a convolutional neural network, we have inputs $x(r)$, where the discrete coordinates $r$ represent palces in a sequence or pixels on a grid, etc.  The **kernel** (sorry, but that's what it's called) function $g(h)$ is chosen so that it goes to zero when $\|h\|$ is too big.  So we started with $x(r)$, and get $t(r) = \sum_{u}{x(r) g(r-u)}$, which is a kind of local average or local weighted sum of the original input features. The kernel function might have some weights or other parameters buried inside it, but if so we apply the _same_ parameters everywhere along the original data.  (We typically hold the "support" of the kernel, the range of values $h$ where $g(h) \neq 0$, fixed during training.)  This means we do the _same_ transformation_ to the original features everywhere.  Convolutional transformations were known already to be a very good way of discriminating textures in images and detecting edges.  The successes of deep-learning image-classifiers came from using multiple layers of convolution.

The result over the last decade has been a flood of work on deep learning,
accompanied, in the popular press, by stories about thinking machines, fears
about technological unemployment, and speculations about how general artificial
intelligence is right around the corner[^cultists].  Some of this results in
extremely impressive and widely-applied technologies, some of it amounts to
re-discovering the wheel (as shown by, e.g.,
@worrying-neural-recommendation-approaches), and some of it poses genuine
unsolved scientific problems.

[^cultists]: So far as I am aware, however, the previous episodes of excitement did not include cults devoted to placating the future AI god.  Whether this counts as progress or not depends, I guess, on how entertaining you find such expressions of the myth-making impulse, and the irony of calling it "rationalism".


# Three Interesting Aspects of Deep Learning

> _This section is long, and optional (in 2022), but hopefully interesting_.

The recent revival of multilayer neural networks has a lot of specific
engineering accomplishments to point to, as well as a lot of more
ambiguous feats of engineering.  But there are also, I think, three specific
_scientifically_ interesting phenomena it has revealed[^names]:

1. **Adversarial examples**, where tiny, humanly-imperceptible perturbations to
  the inputs result in drastic changes to the output;
2. **Interpolation**, where the models generalize to new data (from the
   same distribution) despite _perfectly_ fitting the training data, even
   in cases where the model capacity is high enough to memorize random labels;
3. **Alienness** and **cheating**, where the models achieve high performance at
   matching human classification despite evidently using features which are
   very different from the ones human beings attend to, and often
   getting to high performance by using features that seem to us like accidents
   or tricks.
   
   
[^names]: "Adversarial examples" and "interpolation" are pretty standard names in the literature; "alienness" and "cheating" are not, but there isn't a good single phrase for that phenomenon.

It is not clear how, or whether, these phenomena are related.  It's not even
clear how much they are specific to deep neural networks, as opposed to (say)
highly-parameterized systems trained by gradient descent, or (say) any sort of
high-dimensional classifier with good accuracy over many classes.

## Adversarial Examples

were first noticed, and named, by @intriguing-properties; the phenomenon is
simply illustrated with one of their examples.  Here are some images, from
the standard "ImageNet" dataset [@ImageNet], which were all correctly classified, with
very high confidence, by a then-leading system:

![](intriguing-properties-originals.png)

Here are (believe it or not) _different_ images, which were all classified
as (believe it or not) ostriches:

![](intriguing-properties-adversarials.png)

Here is the _difference_ between the two images, exaggerated by a factor of
10:


![](intriguing-properties-diff.png)

This isn't just a bizarre glitch where this one system happens to do badly on
this one image.  It turns out that we can usually take those
correctly-classified images, and by adding a humanly-imperceptible
perturbation, get something which is, with high probability, classified as
belonging to basically any class we want.

Now you might suppose that what's going on here is that image are very
high-dimensional, but $n$ for the ImageNet dataset is _merely_ 3.2 million, so
the classifier learns to do well on a little "island" around each image but has
no idea what to do when we step off one, and so it says something basically
random.  But if that were true then these classifiers should be vulnerable to
random noise, too.  Yet adding a _larger_ amount of IID Gaussian noise
_doesn't_ send things haywire in the same way [@intriguing-properties,
pp. 6--7].

In computer science, people often talk about "the Adversary" deliberately
crafting inputs or examples to get bad performance out of a method or
algorithm.  It's a vivid way of reasoning about how badly things can go wrong,
and ultimately of obtaining worst-case performance guarantees for
algorithms[^adversary].  @intriguing-properties called their carefully-tweaked
images "adversarial examples" because they're what an Adversary might do to
fool the classifier --- find the smallest perturbation which achieves a 
desired wrong label with at least a specified confidence.  The name has stuck.

[^adversary]: The rhetoric might owe something to the lingering influence of @Wiener-god-and-golem, and, behind that, the cunning and malicious demon of @Descartes-discourse.


One reason the name has stuck is that adversarial examples turn out to be
ubiquitous in deep neural networks.  You can usually eliminate any
particular one you find, but there are (seemingly) always others.  And it's
not just image classifiers: adversarial examples are also known for text
classification [@Gao-Oates-adversarial-text-classification], audio-to-text and other sorts of audio signal
processing [@Taori-et-al-adversarial-examples-for-automatic-speech-recognition], reinforcement learning algorithms that play games or
control robots [@Adversarial-policies], et cetera.  This ubiquity has, potentially, implications
for deploying these systems in the real world.  The fact that one can _print_
adversarially-perturbed images and have the print-outs fool image classifiers
[@Adversarial-patch] at the very least doesn't _sound_ good for, say, self-driving cars.
But here we should focus on the scientific issues that these examples
raise.  (If nothing else, our understanding of the scientific issues will
shae our assessment of the practical implications.)

The thing to focus on, I think, is that ubiquitous, hard-to-eliminate presence
of these examples.  This suggests that it's worth looking for some very
general, equally-ubiquitous aspect or property of very large neural networks,
operating on high-dimensional inputs, trained by gradient descent, which
explains why adversarial examples exist.  (It _could_ be that there's some very
specific explanation for adversarial examples in every different network, and
those specifics can't be usefully abstracted and generalized, but that'd be a
_lot_ of coincidences.)  It might even be that the relevant explanation isn't
about _neural networks_ specifically, but (say) any sort of high-dimensional
classifier trained by gradient descent, and we just happened to notice the
phenomenon first in deep learning.

Nobody has an explanation for adversarial examples which accounts for all the
known facts and commands general consent among knowledgeable experts.  There
are intriguing suggestions about the role of computational constraints
[@Bubeck-et-al-adversarial-examples-from-computational-constraints], about un-intuitive properties of the geometry of high-dimensional
spaces [@adversarial-examples-with-small-hamming-distance], and, relatedly, about a property of
high-dimensional distributions called "concentration of
measure"[^concentration].  But the truth is we just do not know
why this happens.

[^concentration]: The last of these is worth a footnote.  To illustrate "concentration", consider a radius-1 sphere in $p$ dimensions.  What fraction of its volume is within a distance $\epsilon$ of its surface?  Well, the fraction of its volume which _isn't_ is the fraction of its volume inside the smaller sphere of radius $1-\epsilon$; that fraction has to be $(1-\epsilon)^p$.  So the fraction of volume within $\epsilon$ of the surface is $1-(1-\epsilon)^p$.  But this $\rightarrow 1$ exponentially fast as $p\rightarrow\infty$, no matter how small we make $\epsilon$.  Even though the _exact_ surface of the sphere has volume 0, in high dimensions most of the volume is in fact "concentrated" exponentially close to the surface.  This is true not just for sphere but also for cubes and for pretty much any geometric shape you can easily describe.  Probabilistically, if we have a distribution that's independent over $p$ variables, then unless the distribution is _really_ weird, there's a set of very small volume (like the surface of a sphere) which the probability concentrates on [@Boucheron-Lugosi-Massart-concentration].  More exactly, a distribution is "concentrated" when, if a set has _any_ positive probability, it can be expanded by a very small distance to get a set with at least $1/2$ of the total probability.  This phenomenon of "concentration" is particularly clean for distributions of many independent random variables, but it also holds if the dependence between variables is not too strong [@Kontorovich-Raginsky-concentration].  The suggested relevance of this to adversarial examples is, roughly, that the set of images classified as (say) "trucks" needs to have positive probability, as does the set classified as (say) "pandas".  So there would need to be a very small expansion of the classified-as-cars set with probability $> 1/2$, and a very small expansion of the classified-as-pandas set with probability $> 1/2$, which means that the two expansions need to overlap, and so two small steps take us from trucks to pandas.  If the argument in this form seems sketchy, you're not wrong, but see [@adversarial-examples-as-concentration-of-measure].

### Generative Adversarial Networks

Back in the 1980s and 1990s there was a lot of interest in ideas like
"actor-critic" learning architectures, co-evolving classifiers and sets of
examples, etc.[^coevolve]  With the rise of statistical learning, much of this
work faded into obscurity.  In particular, if your goal is "low risk", and you
think of "risk" as "expected loss under the _same_ distribution", it's hard to
know what to make of the idea of evolving training problems to be hard for your
classifier.

[^coevolve]: I won't give a huge collection of references here, but will just mention @Rosin-Belew-competitive-coevolution as an example of a large literature I happen to have handy a quarter-century later.


The discovery of adversarial examples has nonetheless led to a recent
re-invention of ideas like this, though usually without any recognition of the
precedents.  These are **generative adversarial networks** (GANs), named by
@generative-adversarial-nets.  The idea is to train up _two_ networks, a
generator and a discriminator, say $G$ and $D$.  The job of the discriminator
$D$ is easily described: it gets fed cases which are either from the real data
or are simulation output from the generator, and it needs to say which is
which; one typically uses a log loss, and balances the number of real data
cases against the number of generated ones.  The generator needs to come up
with stuff that fools the classifier, and gets rewarded conversely.

We can think of this as a game with two players, and ask about equilibria,
where neither player can improve their pay-off by changing their strategy on
their own.  (The learning process won't _necessarily_ converge on an
equilibrium, but equilibria will _generally_ be fixed points of the learning
process.)  One equilibrium, emphasized by @generative-adversarial-nets, is when
the distribution produced by $G$ is the same as the distribution of the data,
and the discriminator assigns probability $(1/2, 1/2)$ to every example.  In
that case, the GAN will have learned to mimic the distribution perfectly.  But
there's little reason to think that GAN training will _typically_ approach
this.

## Interpolation, and Generalizing Despite Memorizing

Throughout the course, I have emphasized that merely memorizing the training
data is generally a very bad idea.  Partly this is because the training
examples are of no _intrinsic_ interest, being a product of accidents of
sampling and sheer irreproducible noise in the data-generating process.  But
still more it's because it doesn't, usually, help us to generalize to new data.

Now there was already an important caveat to those warnings, which I know
occurred to some of you: nearest neighbors, kNN with $k=1$, does very little
_other than_ memorize the training data.  But we saw in [Lecture 11](https://www.stat.cmu.edu/~cshalizi/dm/22/lectures/11/lecture-11.pdf) that, under some
reasonable-sounding assumptions about continuity, the asymptotic risk of
nearest neighbors is (at most) twice the risk of the _optimal_ decision rule.
That's asymptotically as $n\rightarrow\infty$, but since it _is_ true in the
limit, it must also be true that at finite $n$, nearest neighbors has _some_
ability to generalize.

So 1NN gives us an example of a learning procedure which (i) always memorizes,
but (ii) can generalize _if_ the data-source has the right
properties[^thornton], so (iii) in-sample performance and true risk can be
incredibly different.

[^thornton]: If the data-generating process is such that the optimal rule always does the _opposite_ of what nearby points do (like a checkerboard...), then nearest neighbors will work poorly.  See @Thornton-trash for an elaboration of this point, and some thought-provoking implications about the limitations of similarity-based learning.

I bring this up in the context of neural networks because it turns out that
very large, "over-parameterized"  neural networks turn out to show the same
combination of (i), (ii) and (iii).  In a series of remarkably simple yet
convincing experiments, @Rethinking-generalization took networks that
were known to do well on standard image-classification data sets, and
_randomly_ re-assigned the class labels of the images, so that there was,
in fact, no connection between the image and the label.  The networks succeeded
in memorizing the new labels, but of course their out-of-sample
performance could be no better than chance.  (This is what 1NN
would do in this situation.)  But, again, with the correct labels,
these same networks trained to memorization _do_ generalize.  So these
architectures have the _capacity_ to memorize noise, but still generalize.
In fact, it is fairly common, in applications, to train large
neural networks to the point of memorization and beyond, and till get
good performance under cross-validation or on testing sets.

_Exactly_ what is going on is not clear.  In addition to 1NN, something analogous to this was already known to happen with boosting (which we looked at in [Lecture 13](https://www.stat.cmu.edu/~cshalizi/dm/22/lectures/13/lecture-13.pdf)), where the
most plausible (though not certain) explanation was that boosting
increases the margin, and high-margin classification implies
generalization [@Schapire-et-al-on-boosting].  Spurred by trying to understand
these puzzles, it has _also_ turned out that something similar can happen
with high-dimensional linear models.

To appreciate this last point, imagine we're doing linear regression of $Y$
on $\vec{X}$, but $\dim{\vec{X}} = p > n$.  The OLS estimate of the regression
coefficients requires inverting $\mathbf{x}^T\mathbf{x}$, but this inverse
doesn't exit when $p > n$.  So let's back up a step in OLS's chain of logic.
Minimizing the MSE gives us an equation for $\widehat{\mathbf{\beta}}$,
\[
\mathbf{x}^T \mathbf{x} \widehat{\mathbf{\beta}} = \mathbf{x}^T \mathbf{y}
\]
and the impulse is of course to get $\widehat{\mathbf{\beta}}$ by itself
by multiplying both sides with $(\mathbf{x}^T\mathbf{x})^{-1}$ from the
left.  Again, we can't do that when the inverse doesn't exist, but that
doesn't mean that the equation for $\widehat{\mathbf{\beta}}$ doesn't
have solutions.  Rather, what happens is that there are _infinitely many_
solutions[^collinearity].

[^collinearity]: Similarly, even if $p < n$, if two (or more) columns of $\mathbf{x}$ are linearly dependent, i.e., there is collinearity among the regressors, the difficulty isn't that there are no solutions to $\widehat{\mathbf{\beta}}$.  The difficulty is that we can "trade off" coefficients among the linearly-related regressors, without changing any of the predictions, so there are actually infinitely many equally good coefficient vectors.

When there are infinitely many equally good potential solutions, we're free
to pick one however we like, and many people have thought it sounded sensible
to pick the solution with the smallest norm, the imaginatively-named **min-norm solution**:
\[
\tilde{\beta} = \argmin_{b: \mathbf{x}^T\mathbf{x} b = \mathbf{x}^T \mathbf{y}}{\|b\|}
\]
That is, we take the shortest vector which solves the equation.  If there's
a unique $\hat{\beta}$, then of course $\tilde{\beta} = \hat{\beta}$, but in
the high-dimensional case this still gives us a well-behaved prescription
for finding _a_ coefficient vector (see exercises).

For many, many distributions, if increase the dimension $p$ of $\vec{X}$ but
use the min-norm $\tilde{\beta}$, we get a generalization error that
_decreases_ in $p$ once $p>n$.  It won't always, but it _can_.

Now, in regression, it is _always_ true that
\begin{eqnarray}
\Expect{(Y-\hat{\mu}(X))^2|X=x} & = & \Var{Y|X=x} + \left(\Expect{(\mu(x) - \hat{\mu}(x)}\right)^2 + \Var{\hat{\mu}(x)}\\
& = & (\text{system noise}) + (\text{squared bias of estimate}) + (\text{variance of estimate})
\end{eqnarray}
In these over-parameterized regimes, we've driven bias towards zero.  But,
under these distributions of data, the variance of the estimate is also not
exploding.  What's happening with high-dimensional linear models, for instance,
is that selecting the _minimum_ norm solution is, implicitly, imposing stronger
and stronger constraints on that norm as $p$ grows.  (It's very close to doing
ridge regression and increasing the strength of the penalty as $p$ grows.)

Many people _suspect_ that something like this is at work with deep neural
networks.  That is, some combination of their architecture and their training process meshes with the kind of tasks we give them, so that some implicit
regularization happening on _those_ problems, even though they _could_ memorize any old random noise.  But _exactly_ what's going on here, or how we could characterize it to know when neural networks will be reliable, is still unknown [@Belkin-fit-without-fear].

## Alien Cheaters, or, What the Frog's Eye Tells the Frog's Brain, or, What Is It Like to Be a Convolutional Neural Network?

Here is an image that a leading-in-2015 neural network confidently classifies as
a cheetah:

![](cheetah-from-imagenet.jpeg)

Here is another image which the same neural network is confident is also a cheetah [@deep-neural-networks-are-easily-fooled]:

![](cheetah-fooling.png)

If you look at the second image and, without prompting, also see a cheetah,
you are a very unusual primate and your brain should be promptly (but humanely!) studied by neuropsychologists for the good of science.  For most of us, the second image looks nothing like a cheetah.

Now, the network in question, like almost all the others, was trained to
imitate human judgments and evaluations on standard image repositories[^imagenettoimagenet].
Those systems manage to do this successfully.  But when we step outside
those data sets, it becomes clear that the way the systems are processing
images must be very different from what we East African Plains Apes do.  The scientific questions are thus:

- _what_ features do these systems use,
- _why_ do they use _those_ features, and
- _is_ using those features a good idea?

To put the last point a little science-fictionally[^campbell], have we
(unintentionally) invented devices which process the world in an alien but
equally effective way, or have we made something superficially impressive but
ultimately stupid?

[^imagenettoimagenet]: The photograph data sets were mostly collected by scraping the Web.  @does-imagenet-generalize-to-imagenet went through the interesting exercise of trying to replicate the best-known of these, ImageNet, by repeating the published data-collection protocol.  (Even that was hard.)  They then examined the performance of various published neural networks trained on the original ImageNet on their replication data.  Unsurprisingly, all of them did worse on the replication did.  Very surprisingly (at least to me), the increase in average loss was about the same for all systems!

[^campbell]: John Campbell (1910--1971), an influential early editor of science  fiction, was supposed to tell his authors "Write me a story about a creature that thinks _as well_ as a man, but not _like_ a man".  (You will not be the first to suggest simply writing a story about a woman.)

To dive into this, let's start by examining how the second image above was
created.  @deep-neural-networks-are-easily-fooled began with a random image,
and then optimizing the predicted probability of belonging to the desired
class.  (Specifically, they used an "evolutionary" optimization algorithm,
keeping multiple candidates around, making random "mutations" to them, and copying the ones which did the best job of fooling the image-classification
network.)  What this tells us is that there is a region (or set of regions) in image space
where the classification network assigns very high confidence (say, $> 99.6$%) to the image
being a cheetah (or whatever). This region includes actual pictures of cheetahs, but also certain images of video snow, weird blobs, etc., etc.  If we were to ask human beings to classify images the same way, the region where a human
subject would say "yes, that's definitely a cheetah" would have some overlap with the machine, around the actual pictures, but _not_ in the wilder regions
of the image space.  On the other hand, normal
human beings from my culture immediately recognize these as depictions
of cheetahs, though it's not at all clear a neural network trained on a collection
of _photographs_ would do so:

![http://publicdomainvectors.org/photos/cheetah-cartoon-publicdomain.jpg](cheetah-cartoon-publicdomain.jpeg)

![https://publicdomainvectors.org/photos/CHEETAH.png](CHEETAH.png)


![https://www.publicdomainpictures.net/pictures/450000/velka/image-16505217656FP.jpg](velka.jpg)

This suggests two plausible conclusions:

1. Human beings and current neural networks classify based on very different sets of visual features (or at least using very different weightings on whatever features might be common to both);
2. Human beings and neural networks both give high-confidence classifications to many images which objectively, on a pixel-by-pixel level, are _very_ different from prototypical photographs of members of the class.

Point (2) is particularly suggestive: maybe we're looking at a general property
of classifiers in high-dimensional spaces, or something like that.  But this
is, currently, even less well understood than the existence of adversarial
examples.  In this context, though, adversarial examples are another
indication that whatever features neural networks use are very different from
the features humans are sensitive to, since objectively-small and
humanly-indistinguishable changes clearly move images across neural network
classification boundaries.

The mere fact that neural networks use very different visual features than we
do isn't necessarily bad.  At least, it's bad if we want to use these neural
networks as models of human visual perception, but not necessarily bad as
_engineering_.  (A car is a very bad model of a horse, but it doesn't run any
worse on that account, and making cars move more like horses wouldn't improve
the driving experience.)  But there are very worrying indications that, at
least as currently designed and trained, neural networks end up using visual
features which are, in various ways, bad for their intended applications.

Here[^shane] is an image classified[^caption] by a production system as "sheep":

![https://www.aiweirdness.com/content/images/public/images/e3c62683-f530-49cf-aed2-c2868855f656_2000x1364.jpg](aiweirdness.jpeg)

[^shane]: I learned of this example from Janelle Shane, at [https://www.aiweirdness.com/do-neural-nets-dream-of-electric-18-03-02/], which provides more informative mis-classifications; I strongly recommend Shane's site if the material in these notes is at all interesting.

[^caption]: Actually, as you can read in the image, this system doesn't just classify, it provides automatically-generated captions and tags.  This is _basically_ an elaboration on classification.  (It's a little more complicated; the neural network has learned to associate bits of text with images by mapping them to a common latent space.)

There are, indeed, no sheep here.  But in training sets, there are also
few (no?) pictures of green mossy mountain scenes _without_ sheep (or at least
other animals) for visual interest.  Whatever features the neural network
uses links this image to others in the training set, and it spits out the appropriate classification for them.  It's learned to associate the label "sheep" not
with the animals, or even (to be cautious) photographic traces of animals, but
with certain visual features which were, in the training set, statistically-reliable _signs_ of the animals[^signs].

[^signs]: The connection between signs and probability is very old, and well-explored in @Hacking-emergence.

There is good reason to believe that in this case the signs are things like the text of the meadows.  I mentioned earlier that modern image classifiers
rely heavily on "convolutional" network layers, which apply the same transformation locally over the whole of the image.  Convolutional filters are very good at
detecting rather local features of images like texture, but are intrinsically
ill-suited to notice more global properties.  Take an image, chop it into little squares, shuffle the squares around, and we get something human beings
see as very weird (and perhaps turn into a jig-saw puzzle); a convolutional neural network will however "see" very little difference[^texture].  If it's learned to associate one label with "lots of fuzzy green", well, there's still a lot of fuzzy green...

[^texture]: There are many papers documenting the importance of texture for convolutional neural networks, but I will just mention @Geirhos-et-al-bias-towards-texture and @Brendel-et-al-bag-of-features-vs-deep-learning (the latter did a version of the chop-into-chunks-and-shuffle experiment).

If this was just one goof, well, who cares, that's only a mildly-amusing
anecdote.  More serious is @Carter-et-al-overinterpretation, which explored the issue of what features image
classifiers really use by seeing how much of an image could be "masked" without 
altering classification performance.  Results on leading systems looked like this:

![](imagenet_sis_maintext.png)

That is, the images are being classified correctly (with at least 90% confidence) using only thin strips at the borders.  Again, this is enough to get at predominant textures, which seem to be most of what this (convolution-reliant) system uses.  But this should be very worrisome when we consider any sort of real-world use.
Even if we're not worried about someone deliberately trying to fool the
system[^worriedaboutfooling], the sheer, inhuman strangeness of the system ought tio make us doubt that we can anticipate how it will work in the wild.

[^worriedaboutfooling]: But why wouldn't we be?  If the system is worth building for a real-world application, it'll also be worth somebody's time to try to subvert.  "What gets measured gets manipulated" and "Some people cheat" are two very sound design principles...

Now in fact there is _very_ good evidence that neural networks frequently seize
upon features which are predictive in their training context, but _only_ in
their training context, in "high-stakes" situations.  A compelling (even
disturbing) one comes from @deep-learning-learns-hospitals-and-x-ray-machines,
which re-examined the ability of neural networks to diagnose diseases from
chest x-rays.  It turned out that x-rays from different diseases had been
(predominantly) collected using different types of machines (portable
vs. fixed) and at different hospitals; that the machine and hospital
information was printed on the x-ray image; and that the neural network had
learned, in training, to give a lot of weight to _that_ part of the image. 
So the neural network learned to be accurate, but in a useless way.

Or, more exactly, in a way which will continue to be accurate only if the
environment in which the neural network is used continues to work the same way
as the training environment.  If something (other than the neural network!)
continues shunting patients with different diseases to different hospitals, and
hospital information keeps being added to x-ray images in the same way as
before, the neural network will keep working just fine.  The doctors asked for
a system to do diagnosis (a real-world problem).  The data analysts translated
that into a tractable statistical problem: come up with a learning process
which will find a decision rule with low risk in _that_ environment.  They
solved that problem, they ran through the learning process, and they got a rule
which delivered _exactly_ on that promise.  The snag is that "diagnose
reliably" is _not_ the same as "classify accurately in a given environment".
If we are unhappy that the rule doesn't also have low risk in other, different
environments that the learning process was never exposed to, we have no one to
blame but ourselves.

It should, perhaps, not be surprising that we see behavior like this from
neural networks.  One of the classic early papers about biological neural
computation was titled "What the frog's eye tells the frog's brain"
[@what-the-frogs-eye-tells-the-frogs-brain].  By very careful experimentation,
they showd that the frog's retinas contained neural networks which detected the
presence, location and velocity of small dark rapidly-moving blobs, and
transmitted that information along the optic nerve to the frog's brain.  In the
frog's ecological context, that was information about flies and other insects
the frog could catch and eat with its tongue.  In the lab, those small dark
jittery blobs could be all kinds of things, and in environment with (say) many
other little dark flying bits (gravel? metal shavings?) this part of the
percentual system mightn't work very well for a hungry frog, or at least might
need to be supplemented with other information.

We have developed neural networks which (at their best) work like _that_.  They
are very little like intelligence, as we experience conscious rational thought,
or even the forms of unconscious thought underlying (say) grammar.  Rather,
they are much closer to unconscious, automatic perception.  Experience and
evolution has tuned our perceptions to work well in a certain range of
environments, it is (as the psychologist Gerd Gigerenzer puts it) "ecological
rational".  But _outside_ those environments, our automatic perceptions can go
horribly awry.

Some would argue that a great deal of human intelligence is also only "ecologically rational".  (Gigerenzer, for one, makes a strong case for this [@Gigerenzer-adaptive-thinking].)  We look at the mistakes of a neural network and laugh, or swear not to use something so stupid.  But our own stubborn visual illusions seem to us mere quirks
which in no way detract from our dignity as rational creatures.  Perhaps we're
right about this; perhaps this is mere chauvinism, and a truly alien
intelligence, "vast, cool, and unsympathetic" (as the poet says), would
draw no distinction between us and our artifacts[^futurity].

[^futurity]: One might even imagine those vast, cool, unsympathetic intellects being, in some distant future, remote descendants of our current neural networks, carrying pattern-recognition modules optimized for ImageNet around as traces of an evolutionary past as distant for them as our own blood's mimicy of the chemistry of prehistoric seas.  But I imprinted on Arthur C. Clarke as a boy, and am launching a flight of fancy, and we should return to our actual machines.

There is however one straightforward way in which it is bad for neural networks
to use weird-to-us features.  We want to deploy those neural networks in _our_
world, in a physical and social environment we have grown up in and have built
to work with _our_ perceptions, modes of thought and bodies.  If our machines
do not work well in _that_ range of environments, they are badly engineered
_for us_, and we should designed systems which work better _for us_.  The
alternative is to redesign our world for the convenience of our machines.
That seems both stupid, and inconsistent with "the human use of human
beings" [@Wiener-human] --- which doesn't mean it won't be tried.

Before closing this section, I want to mention two things.  First, it is very
plausible that there is some connection between this issue, that of using
strange and apparently fragile features, and the other issues I highlighted,
those of adversarial examples and interpolative generalization.  Plausible,
but by no means established!  Second, it is, again, quite
unclear how far this issue is peculiar to deep neural networks.  If anyone has tried to
comparable studies on, say, support vector machines or random forests, I
haven't seen it.  (I haven't tried to do it myself!)  But it might be that we're looking at phenomena which are fairly generic to large-scale learning
systems, rather than specific to neural networks.




# Further reading

@Deep-Learning-book is, deservedly, the standard text on deep learning, though
algorithmic details advance rapidly, and are hard even for specialists to keep
up with.

From the last burst of interest in neural networks, @Churchland-Sejnowski is
still good, and recently brought back into print.  @Ripley-pattern-recognition
is full of good ideas about statistical applications, and has the unusual
distinction of being written by a statistician deeply involved in the
development of R.

On distributed representations, @Abbott-Sejnowski-neural-codes is a now-classic
volume of important, and generally readable, papers.  @Churchland-Sejnowski
is also a good reference on this topic.  That the meaning of the activity
of neural networks might only reside in the global configuration, and not
in individual cells, was more or less explicit already in the work of
some of the pioneers, such as @Hayek-sensory-order and @Hebb-organization
(and, at a different level of biological organization, @Luria-working-brain).

As discussed above, our understanding of how models with the capacity to
memorize noise can also generalize to new data is still very incomplete.  I
recommend the review/discussion paper by @Belkin-fit-without-fear, written by
one of the leaders in the field and with excellent references to earlier work.
(If this intrigues you, you might also try your hand at the exercises.)



# Exercises

I recognize that it's a bit perverse that all the exercises in notes on neural
networks relate to over-parameterized linear models.

1. _Min-norm linear regression, part 1_.  Consider the ordinary-least squares problem $\mathbf{x}^T\mathbf{x} \vec{b} = \mathbf{x}^T \mathbb{y}$, where $p > n$.  Let's say that any vector $\vec{b}$ which satisfies this equation is an "OLS solution".  Recall (from linear algebra) that the collection of vectors $\vec{v}$ where $\mathbf{a}\vec{v} = 0$ is called the **null space** of the square matrix $\mathbf{a}$.
    a. Show that if $\vec{v}_1$ and $\vec{v}_2$ are both in the null space of $\mathbf{a}$, then $c_1 \vec{v}_1 + c_2 \vec{v}_2$ is also in the null space.  (This justifies the "space" part of the name "null space".)
    b. Show that the dimension of the null space of $\mathbf{x}^T\mathbf{x}$ is at least $p-n$.  (Some people call this number the "nullity" of the matrix.)  Can you given an example of an $\mathbf{x}$ where the dimension of the null space is larger than $p-n$?  (We're assuming $p>n$ here so don't try $p=1$, $n=1000$ or anything silly like that.)
    c. Show that if $\vec{b}$ is an OLS solution, and $\vec{v}$ is in the null space of $\mathbf{x}^T\mathbf{x}$, then $\vec{b} + \vec{v}$ is also an OLS solution.  Use this to explain why the space of OLS solutions has dimension $p-n$ (at least).
    d. Show that the vector at the origin of $p$-dimensional space, $\vec{0}$, has a unique projection on to the space of OLS solutions.  Explain why the norm of the projection of $\vec{0}$ is the minimum value of the norm in the whole space of OLS solutions.  _Hint_: for any vector $\vec{v}$, $\|\vec{v}\| = \mathrm{distance}(\vec{v}, \vec{0}) = \mathrm{distance}(\vec{0}, \vec{v})$.  (Why?)
2. _Min-norm linear regression, part 2_. Suppose that $p > n$, and we look for
the minimum norm solution:
\[
\tilde{\beta} = \argmin_{b: \mathbf{x}^T\mathbf{x} b = \mathbf{x}^T \mathbf{y}}{\|b\|}
\]
    a. Explain why
\[
\tilde{\beta} = \argmin_{b: \mathbf{x} b = \mathbf{y}}{\|b\|}
\]
   b. (Harder) Explain why
\[
\tilde{\beta} = \mathbf{x}^T(\mathbf{x}\mathbf{x}^T)^{-1}\mathbf{y}
\]
The quantity $\mathbf{x}^T(\mathbf{x}\mathbf{x}^T)^{-1}$ is often written as $\mathbf{x}^{\dagger}$, and called[^mppi] the **Moore-Penrose pseudo-inverse**, or **Moore-Penrose generalized inverse**, of $\mathbf{x}$.
3. _Interpolation in the linear model and risk_.  Consider the situation where $Y= Z + \epsilon$, $Z \sim \mathrm{Unif}(-10,10)$ and $\epsilon \sim \mathcal{N}(0,1)$.  However, we do not get to see $Z$, but instead get to observe $\vec{X}$ where $\vec{X}_j = \sin{(\omega_j Z + \phi_j)}$.  The frequencies $\omega_j$ are also $\mathcal{N}(0,1)$, and the phases $\phi_j$ are $\sim\mathrm{Unif}(0,2\pi)$.  The coordinates of $\vec{X}$ are thus random Fourier features, as in a previous lecture.  Notice that while there is a simple relationship between $Y$ and $Z$, there is no _exact_ linear relationship between $Y$ and $\vec{X}$, though one can come increasingly close by using more and more random features, say $d$.
    a. Write a function which will generate $n$ IID $(Z,Y)$ pairs from the specification above.  It should return a two-column data frame.
    b. Write a function which takes in a two-column, $n$-row data $(Z, Y)$ data frame, as from (a), and a two-column, $d$-row array (or data frame) of $\omega_j$ and $\phi_j$, and returns the $(d+1)$-column, $n$-row data frame of $(\vec{X}, Y)$ pairs.
    c. Write a function which takes a data frame as returned by (b), and a vector of $d$ coefficients, and returns the MSE of using those coefficients, applied to the $\vec{X}$ columns, to predict the $\vec{Y}$ column.  (You can assume the intercept term is zero.)
    d. Write code which, given a data frame as returned by (c), _either_ finds the OLS coefficients, if $n > d$, _or_ finds the min-norm solution.  In doing the latter, you may find the `ginv` function from the `MASS` package helpful.
    e. Generate a training set with $n=100$ and $d=1000$.  (Make sure to save the $\omega, \phi$ values for the features in their own array.)  Calculate the in-sample MSE of linear models fitted to the first $p$ features, for $p \in 1:d$.
    f. Generate an evaluation set with $n=100000$ and the same $d=1000$ random Fourier features.  For each $p\in 1:d$, calculating the MSE of linear models fitted to the first $p$ features on the training set on this evaluation set.
	g. Plot in-sample and generalization MSE as a function of $p$.  What happens?
	h. Plot the ($L_2$) norm of the estimated coefficients vectors as a function of $p$.  What happens?
	

[^mppi]: More pedantically, the Moore-Penrose pseudo-inverse of a $n\times p$ matrix $\mathbf{a}$ is the (unique) $p\times n$ matrix $\mathbf{a}^{\dagger}$ such that $\mathbf{a}\mathbf{a}^{\dagger}\mathbf{a} = \mathbf{a}$ and $\mathbf{a}^{\dagger}\mathbf{a}\mathbf{a}^{\dagger}=\mathbf{a}^{\dagger}$ (so $\mathbf{a}$ and $\mathbf{a}^{\dagger}$ "cancel out" in matrix products), and where $\mathbf{a}\mathbf{a}^{\dagger}$ and $\mathbf{a}^{\dagger}\mathbf{a}$ are both symmetric.  If $p=n$ and $\mathbf{a}$ is invertible, then $\mathbf{a}^{\dagger}=\mathbf{a}^{-1}$.  If $n< p$ but $\mathbf{a}$ has full rank (=linearly-independent rows), then $\mathbf{a}^{\dagger} = \mathbf{a}^{T}(\mathbf{a}\mathbf{a}^T)^{-1}$.  If $n > p$ but $\mathbf{a}$ has full rank (=linearly-independent columns), then $\mathbf{a}^{\dagger}=(\mathbf{a}^T\mathbf{a})^{-1}\mathbf{a}^T$, which should look familiar.


# References

