---
title: From Rademacher complexity to VC dimension
output: slidy_presentation
date: 4 March 2021 (Lecture 9)
author: 36-465/665, Spring 2021
bibliography: locusts.bib
---

\newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)}
\newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]}
\newcommand{\Risk}{r}
\newcommand{\EmpRisk}{\hat{r}}
\newcommand{\Loss}{\ell}
\newcommand{\OptimalStrategy}{\sigma}
\DeclareMathOperator*{\argmin}{argmin}
\newcommand{\ModelClass}{S}
\newcommand{\OptimalModel}{s^*}
\DeclareMathOperator{\tr}{tr}
\newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}}
\newcommand{\myexp}[1]{\exp{\left( #1 \right)}}
\newcommand{\eqdist}{\stackrel{d}{=}}
\newcommand{\Rademacher}{\mathcal{R}}
\newcommand{\EmpRademacher}{\hat{\Rademacher}}
\newcommand{\Growth}{\Pi}
\newcommand{\VCD}{\mathrm{VCdim}}


## Previously

- Problem: We want to control $\Risk(\hat{s}) - \EmpRisk(\hat{s})$ (over-fitting,
optimism) and $\Risk(\hat{s}) - \Risk(s^*)$ (estimation error, excess risk,
distance to optimum)
- [Solution](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/07/lecture-07.html): $\Gamma_n = \max_{s}{|\Risk(s) - \EmpRisk(s)|}$ and especially $\Expect{\Gamma_n}$
- Problem: expected maxima are hard to calculate, and doesn't it look like we'd need to  know $\Risk(s)$ anyway?
- [Solution](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/08/lecture-08.html): Rademacher complexity $\Rademacher_n \equiv \Expect{n^{-1}\max_{s \in S}{|\sum_{i=1}^{n}{\sigma_i \Loss(Y_i, s(X_i))}|}}$ because $\Expect{\Gamma_n} \leq 2 \Rademacher_n$
    + "How well could we (seem to) correlate with coin tossing?" tells us about how much we might be over-fitting
- Problem: still the expectation of a maximum


## Three ways forward from Rademacher complexity

1. _Theory_: calculations involving the distribution and the models
2. _Data-dependence_: Looking at how the models actually perform over the data
3. _Distribution-independence_: Using properties of the models that hold in any distribution or data set


## Theory

- Sometimes we can do probability theory to get at the expectation, or at
least bound it
- One example [@Mohri-Rostamizadeh-Talwalkar, theorem 4.2, p. 77]:

> Suppose our $\Prob{\|X\| \leq r} = 1$, and we're using linear models, so $s(x) = x\cdot \beta$, with $\| \beta \| \leq b$.  Then $\EmpRademacher_n \leq \frac{rb}{\sqrt{n}}$, so the same bound holds for $\Rademacher_n$

- (We will come back to this particular example later in the course)



## Data-dependent bounds {.smaller}

- The basic Rademacher complexity [bound on generalization error](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/08/lecture-08.html#(11)) is (for bounded loss $0\leq \Loss \leq m$)
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + 2\Rademacher_n + m \sqrt{\frac{\log{1/\alpha}}{2n}}} \leq \alpha
\]
- This is a **distribution-dependent** bound: $\Rademacher$ is an expectation so it changes with the distribution of $(X,Y)$
- One route forward: go from a distribution-dependent bound to a **data-dependent** bound that involves a random quantity we calculate from the data, [e.g.](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/08/lecture-08.html#(13))
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + 2\EmpRademacher_n + 3m\sqrt{\frac{\log{2/\alpha}}{2n}}} \leq \alpha
\]
or even
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + 2\hat{\EmpRademacher}_n + \text{spoiler for HW5Q1}} \leq \alpha
\]
    + Data-dependent bounds don't _have_ to involve Rademacher complexity (e.g. @Lunde-CRS-bootstrap-GEB gives one using the bootstrap)


## Distribution-free bounds {.smaller}

- Look at the definition of Rademacher complexity again, for a general class of functions $\mathcal{F}$
\[
\Rademacher_n \equiv \Expect{\max_{f \in \mathcal{F}}{\frac{1}{n}\left|\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|}}
\]
- This is "on average, how well could our functions seem to correlate with binary noise?"
- Distribution-dependent because of the "on average"
- If we could say "how well could our functions _possible_ seem to correlate with noise?" we'd get something **distribution-free**
- A **distribution-free generalization error bound** is one where
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + g(n,\alpha, S)} \leq \alpha
\]
and $g(n, \alpha, S)$ is deterministic _and_ the same for _all_ data-generating distributions
    + and $g(n, \alpha, S) \rightarrow 0$ as $n\rightarrow\infty$


## Getting a distribution-free bound for classification

- Assume $s(x) = \pm 1$ for all $x$, and use $0-1$ loss
    + (Means that $\Rademacher$ of loss-and-model is $1/2$ of $\Rademacher$ of model)
- So $\sigma_i s(X_i) = 1$ if the signs agree and $=-1$ if the signs disagree
- So $n^{-1} \sum_{i=1}^{n}{\sigma_i s(x_i)} = 1$ when all the signs agree, and $\approx 0$ if $\sigma$ and $s(X_i)$ are independent
    + More exactly under independence $n^{-1} \sum_{i}{\sigma_i s(X_i)}$ is random with mean zero and standard deviation $O(1/\sqrt{n})$
- So Rademacher complexity will be high if, no matter what the random signs $\sigma_1, \ldots \sigma_n$ are, we can always find _some_ $s \in S$ where $s(x_1), \ldots s(x_n)$ have the same signs
- Turn this around: if $S$ _can't_ always produce every possible classification, that will limit the Rademacher complexity


## Growth function

- Continue with $s(x) = \pm 1$ assumption
- The **growth function** of $S$, $\Growth_{S}(n)$, is the maximum number of
ways we can classify $n$ points $x_1, \ldots x_n$ using functions from $s$:
\[
\Growth_{S}(n) \equiv \max_{x_1, \ldots x_n}{\# \left\{ (s(x_1), \ldots s(x_n)) ~:~ s \in S\right\}}
\]
- Notice: nothing about distributions in here, just properties of the model class $S$
- Obvious bound: $\Growth_S(n) \leq 2^n$


## The growth function upper-bounds the Rademacher complexity

- Claim:
\[
\Rademacher_n(S) \leq \sqrt{\frac{2 \log{\Growth_{S}(n)}}{n}}
\]
- Proof: HW5
- Implication:
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + \sqrt{\frac{2 \log{\Growth_{S}(n)}}{n}} +  \sqrt{\frac{\log{1/\alpha}}{2n}}} \leq \alpha
\]
- This is our first distribution-free generalization error bound

## It really matters whether the growth function stays exponential

- $\log{\Growth_{S}(n)} \leq n\log{2}$, because $\Growth_{S}(n) = 2^n$
- **If** $\Growth_{S}(n) = 2^n$, then the distribution-free bound becomes
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + \sqrt{2\log{2}} + \sqrt{\frac{\log{1/\alpha}}{2n}}} \leq \alpha
\]
which is not useful at all!
- **If** $\Growth_{S}(n)$ grows at a slower exponential rate than $2^n$, say $1.1^n$, we still get a constant-size generalization error bound (also not useful)
- **If** $\Growth_{S}(n)$ is only polynomial in $n$, then the bound shrinks fairly, like $O(\sqrt{\frac{\log{n}}{n}})$


## Shattering and the VC dimension

- A set $\left\{ x_1, \ldots x_n \right\}$ is **shattered by $S$** if we can get any labeling of the points we like by picking the right $s \in S$
- If $\Growth_{S}(n) = 2^n$, then there is _some_ set of size $n$ which can be shattered by $S$
- If $\Growth_{S}(n) < 2^n$, then there is _no_ set of size $n$ which can be shattered by $S$
- The **Vapnik-Chervonenkis (VC) dimension of $S$** is the largest $n$ for which $\Growth_{S}(n) = 2^n$
- $\VCD(S) =$ size (cardinality) of the largest set shattered by $S$
    + If there is no largest shattered set, $\VCD(S) = \infty$
- Again: VC dimension is entirely about $S$, nothing about the distribution of $X$ or the joint distribution of $(X,Y)$

## Some examples of VC dimension

- Using linear classifiers in the plane, we can shatter any set of 3 points, but not 4 points, so $\VCD = 3$
```{r, echo=FALSE}
par(pty="s")
plot(x=c(-1, 0, 0, 1), y=c(0, -1, +1, 0), pch=c(1, 2, 2, 1), xlab="", ylab="")
```
- More generally: linear classifiers in $d$ dimensions can shatter $d+1$ points but not $d+2$, so $\VCD=d+1$

## Some examples of VC dimension (2)

- Convex polygons with $d$ points: can always shatter up to $2d+1$ points arranged in a circle, but no more
- Sine waves: $x$ is points on the line, say $s(x) = 1$ if $\sin{(\omega x)} \geq 0$ and $s(x) = -1$ if $\sin{(\omega x)} \leq 0$: we can shattered _arbitrarily large_ sets of points by adjusting $\omega$, so $\VCD= \infty$
    + One parameter family  but _infinite_ VC dimension
- People have done this for many, many different kinds of models
    + @Anthony-Bartlett-neural-network-learning and @Vidyasagar-on-learning-and-generalization are two good references for results like this


## Finite VC dimension $\Rightarrow$ distribution-free bounds

- A remarkable fact (proved by V and C with intricate combinatorics):

> If $\VCD(S) = d < \infty$, then for $n \geq d$, \[ \Growth_{S}(n) \leq \left( \frac{en}{d} \right)^d = O(n^d) \] while if $\VCD(S) = \infty$, then $\Growth_{S}(n) = 2^n$ for all $n$

- Growth functions are _either_ exponential at the maximum possible rate, _or_ they are polynomial
- Consequence: for $n \geq d$,
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + \sqrt{\frac{2d\log{\frac{en}{d}}}{n}} + \sqrt{\frac{\log{1/\alpha}}{2n}}} \leq \alpha
\]
- Moral: the optimism/over-fitting $\Risk(\hat{s}) - \EmpRisk(\hat{s})$ is $O(\sqrt{\frac{\log{n/d}}{n/d}})$ (with high probability)
    + So is the excess risk/estimation error $\Risk(\hat{s}) - \Risk(s^*)$


## Note 1: What makes this a "dimension"?

- Area of a square is $r^2$, volume of a cube is $r^3$, volume of a hyper-cube is $r^d$
- Volume : Geometric dimension :: Growth function : VC dimension
- Even closer analogy: say $N(\epsilon)$ is the number of cubes  of side $\epsilon$ needed to cover a geometric figure, then in $d$ dimensions $N(\epsilon) \simeq (1/\epsilon)^{d}$
    + More precisely $\lim_{\epsilon \rightarrow 0}{\frac{\log{N(\epsilon)}}{\log{1/\epsilon}}} = d$ ("box-counting dimension")
- Covering number: Geometric dimension :: Growth function : VC dimension


## Note 2: VC dimension and falsifiability

- @Popper-logic: what makes a hypothesis scientific is if it could be **falsified** by observations (we could possibly tell that it was wrong)
- Consider the hypothesis "$Y = s(X)$ for some $s \in S$"
- If $\VCD(S) < \infty$, then this hypothesis is falsifiable, because if $n > \VCD(S)$, we _could_ see data which doesn't match _any_ $s \in S$
- For you to ponder off-line: if $\VCD(S) = \infty$, is this hypothesis un-falsifiable?


## VC dimension and uniform convergence

> - If $\VCD(S) < \infty$, then $\Gamma_n \rightarrow 0$ in all distributions
>     + Follows from the bounds we've done already
> - Finite VC dimension is _sufficient_ for uniform convergence under all distributions
> - If $\VCD(S) = \infty$, then $\Gamma_n \not\rightarrow 0$ under at least some distributions
>    + Considerably more complicated to prove
> - Finite VC dimension is _necessary_ for uniform convergence under all distributions
> - N.B., there might still be useful distribution-dependent or even data-dependent bounds, just not distribution-free ones


## Beyond binary classifiers with 0-1 loss

- If we're interested in continuous-valued functions, there are ways of
extending the notion of growth function and VC dimension
    + Basically: look at whether $s(x) \leq t$ as we vary $t$, to get binary classifiers from continuous functions (handout on class website)
- The growth function and VC dimension bounds have corresponding results for continuous functions
- Sub-exponential growth function / finite dimension $\Rightarrow$ distribution-free bounds


## Trade-offs {.smaller}

#### Between different bounds

- Distribution-free bounds tend to be looser than distribution-dependent bounds
- Distribution-free bounds tend to match distribution-dependent bounds _for the worst possible distribution_
    + Mythology: the Adversary gets to pick the distribution of $(X,Y)$ after seeing $S$, but doesn't get to alter the random draws from that distribution
	
#### Between different model classes

- Consider two model classes $S_1 \subset S_2$
- The larger model will generally have better optimal risk,
\[
\min_{s \in S_1}{\Risk(s)} \geq \min_{s \in S_2}{\Risk(s)}
\]
- The larger model will generally also have looser bounds, because
\begin{eqnarray}
\Gamma_n(S_1) & \leq & \Gamma_n(S_2)\\
\Rademacher_n(S_1) & \leq & \Rademacher_n(S_2)\\
\Growth_{S_1}(n) & \leq & \Growth_{S_2}(n)\\
\VCD(S_1) & \leq & \VCD(S_2)
\end{eqnarray}
- So there's a trade-off between approximation error and estimation error
    + More complex models give better approximations but are harder to estimate
    + Will revisit when we look at model selection in a few weeks


## Summing up

- Rademacher complexity gives us _distribution-dependent_ bounds on generalization error and excess risk
- Empirical Rademacher complexity, estimated empirical Rademacher complexity, etc., take us to _data-dependent_ bounds
- The growth function takes us from Rademacher complexity to _distribution-free_ bounds
- The growth function is either polynomial in $n$, or $2^n$, depending on whether $\VCD$ is finite or infinite
- Finite $\VCD$ is sufficient for empirical risk minimization to work, and _necessary_ to guarantee it will work for _all_ distributions
- We can figure out $\VCD$ by geometry and/or combinatorics and/or prior work


## References

(Popper's book is actually from 1934 but R Markdown's bibliography processor isn't doesn't understand how to handle translated works)



