---
title: From Convergence to Generalization Error via Uniform Convergence
author: 36-465/665, Spring 2021
date: 25 February 2021 (Lecture 7)
output: slidy_presentation
---

\[
\newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)}
\newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]}
\newcommand{\Risk}{r}
\newcommand{\EmpRisk}{\hat{r}}
\newcommand{\Loss}{\ell}
\newcommand{\OptimalStrategy}{\sigma}
\DeclareMathOperator*{\argmin}{argmin}
\newcommand{\ModelClass}{S}
\newcommand{\OptimalModel}{s^*}
\DeclareMathOperator{\tr}{tr}
\newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}}
\newcommand{\myexp}[1]{\exp{\left( #1 \right)}}
\]


## Previously

- Risk of strategy $s$ is $\Risk(s) \equiv \Expect{\Loss(Y, s(X))}$
- Optimal strategy $\OptimalStrategy \equiv \argmin_{\text{all functions}\ s}{\Risk(s)}$
- Best-in-class $\OptimalModel \equiv \argmin_{s \in \ModelClass}{\Risk(s)}$
- $\Risk(s) = \Risk(\OptimalStrategy) + (\Risk(\OptimalModel) - \Risk(\OptimalStrategy)) + (\Risk(s) - \Risk(\OptimalModel))$
- Empirical risk $\EmpRisk(s) \equiv n^{-1}\sum_{i=1}^{n}{\Loss(Y_i, s(X_i))}$
- Empirical risk minimizer $\hat{s} = \argmin_{s \in \ModelClass}{\EmpRisk(s)}$
    + Best performance, on the training data, within the class $\ModelClass$
- How well does $\hat{s}$ _actually_ do in the future?


## Why we're _almost_ there {.smaller}

- Risk is the expected empirical risk:
\begin{eqnarray}
\Expect{\EmpRisk(s)} & = & \Expect{n^{-1}\sum_{i=1}^{n}{\Loss(Y_i, s(X_i))}}\\
& = & \frac{1}{n}\sum_{i=1}^{n}{\Expect{\Loss(Y_i, s(X_i))}})\\
& = & \frac{1}{n}\sum_{i=1}^{n}{\Risk(s)} ~ (\text{because IID})\\
& = & \Risk(s)
\end{eqnarray}
- We now know lots of ways to bound deviations from expectations:
\[
\Prob{|Z-\Expect{Z}| \geq \epsilon} \leq \alpha_{Z}(n, \epsilon)
\]
    + Function $\alpha$ involves properties of $Z$
- We can apply these to $\Risk$ and $\EmpRisk$, to get
\[
\Prob{|\EmpRisk(s) - \Risk(s)| \geq \epsilon} \leq \alpha_{\EmpRisk(s)}(n, \epsilon)
\]
    + Assume $\Loss$ is bounded, or sub-Gaussian, etc. to get function $\alpha$
    + HW3Q3: invert this to get margin of error at desired confidence level $\alpha$ and sample size $n$, or sample size required given  desired confidence level $\alpha$ and desired margin of error $\epsilon$
	

## Why we're not quite there

- $\hat{s}$ isn't independent of the data:
\begin{eqnarray}
\Expect{\EmpRisk(\hat{s})} & = & \Expect{n^{-1}\sum_{i=1}^{n}{\Loss(Y_i, \hat{s}(X_i))}}\\
& = & \frac{1}{n}\sum_{i=1}^{n}{\Expect{\Loss(Y_i, \hat{s}(X_i))}}
\end{eqnarray}
- Now the terms in the sum _aren't_ IID so this _isn't_ just $\Risk(\hat{s})$
   + Still identically distributed but not independent

## What's the issue, mathematically?

- **Universal** convergence: You can pick any one $s \in \ModelClass$, and any $\epsilon > 0$, and I can always find $\alpha_{s}(n, \epsilon)$ which guarantees that
\[
\Prob{|\EmpRisk(s) - \Risk(s)| \geq \epsilon} \leq \alpha_{s}(n, \epsilon)
\]
with
\[
\lim_{n\rightarrow \infty}{\alpha_{s}(n, \epsilon)} = 0
\]
- Meaning: every $\EmpRisk(s)$ converges to its $\Risk(s)$
- but _at its own rate_ (given by the $\alpha_s$ function)
- Convergence might be really, really slow for some strategies $s$
- which'd be bad news if the _optimal_ strategy $\OptimalModel$ was really
slowly converging


## What's the issue, mathematically? (2)

- **Universal** convergence:  for every $s \in \ModelClass$, you can pick any $\epsilon > 0$, and I can find $\alpha_{s}(n, \epsilon)$ which guarantees that
\[
\Prob{|\EmpRisk(s) - \Risk(s)| \geq \epsilon} \leq \alpha_{s}(n, \epsilon)
\]
with
\[
\lim_{n\rightarrow \infty}{\alpha_{s}(n, \epsilon)} = 0
\]
- Versus **uniform** convergence: you can pick any $\epsilon > 0$, and I can find a _single_ $\alpha(n, \epsilon)$ so that
\[
\Prob{\max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \geq \epsilon} \leq \alpha(n, \epsilon)
\]
with
\[
\lim_{n\rightarrow\infty}{\alpha(n, \epsilon)} = 0
\]
- _All_ the strategies converge _at least_ that fast
    + Including the optimal strategy $\OptimalModel$
	

## How would uniform convergence help?

- Let's abbreviate:
\[
\Gamma_n \equiv \max_{s \in S}{|\EmpRisk(s) - \Risk(s)|}
\]
- Uniform convergence means $\Gamma_n \rightarrow 0$ as $n\rightarrow\infty$
- _Empirical risk vs. true risk_:
\[
|\EmpRisk(\hat{s}) - \Risk(\hat{s})| \leq \max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} = \Gamma_n
\]
- $\Rightarrow$ Knowing how fast $\Gamma_n$ goes to 0 tells us how optimistic the empirical risk is


## How would uniform convergence help? (2)

- _True risk vs. optimal true risk_:
\begin{eqnarray}
\Risk(\hat{s}) - \Risk(\OptimalModel) & = & \Risk(\hat{s}) - \EmpRisk(\OptimalModel) + \EmpRisk(\OptimalModel) - \Risk(\OptimalModel)\\
& \leq & \Risk(\hat{s}) - \EmpRisk(\hat{s}) + \EmpRisk(\OptimalModel) - \Risk(\OptimalModel)\\
& \leq & |\Risk(\hat{s}) - \EmpRisk(\hat{s})| + |\EmpRisk(\OptimalModel) - \Risk(\OptimalModel)|\\
& \leq & 2\max_{s}{|\EmpRisk(s) - \Risk(s)|} = 2\Gamma_n
\end{eqnarray}
    + Works because $\EmpRisk(\hat{s}) \equiv \min_{s \in \ModelClass}{\EmpRisk(s)} \leq \EmpRisk(\OptimalModel)$
- $\therefore$ Knowing how fast $\Gamma_n \rightarrow 0$ tells us how close the empirical risk minimizer $\hat{s}$ will be to the best-in-class strategy $\OptimalModel$


## How could we get uniform convergence?

- Suppose the class of strategies $\ModelClass$ was _finite_:
\[
S = \left\{ s_1, s_2, \ldots s_q \right\}
\]
with $q < \infty$
- Then universal convergence $\Rightarrow$ uniform convergence:
\begin{eqnarray}
\left\{\max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \geq \epsilon \right\} & = & \left\{ \exists i \in 1:q ~\text{such that} ~ |\EmpRisk(s_i) - \Risk(s_i)| \geq \epsilon \right\}\\
\left\{\max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \geq \epsilon \right\} & = & \cup_{i=1}^{q}{\left\{ |\EmpRisk(s_i) - \Risk(s_i)| \geq \epsilon \right\}}\\
\Prob{\Gamma_n \geq \epsilon} & \leq & \sum_{i=1}^{n}{\Prob{|\EmpRisk(s_i) - \Risk(s_i)| \geq \epsilon}}\\
& \leq & q \max_{i \in 1:q}{\Prob{|\EmpRisk(s_i) - \Risk(s_i)| \geq \epsilon}}
\end{eqnarray}
- This is called the **union bound**
    + For any two events $A$ and $B$, $\Prob{A \cup B} = \Prob{A} + \Prob{B} - \Prob{A \cap B} \leq \Prob{A} + \Prob{B}$
- Extra simple when all the deviation probabilities on the RHS are the same
    + E.g. if the loss function $\Loss$ is bounded


## Why the union bound isn't enough

- Most interesting strategy classes are infinite!
- $\therefore$ union bound tells us the deviation probability for $\Gamma_n < \infty$ which is not helpful
- Two ways to rescue this:
    1. Approximating an infinite $\ModelClass$ by a _finite_ one, then union bound + approximation inaccuracy
	2. Trickery: directly calculate (or bound) $\Expect{\Gamma_n}$ and then argue $\Gamma_n$ concentrates on its expected value


## The route of approximation

- Fix $\epsilon > 0$
- Suppose we can find $q(\epsilon)$ strategies $s_1, \ldots s_q$ which approximate all the $s$'s to within $\pm \epsilon$:
    + You get to name any $s$ you like, and I find an $s_i$ in the set where, no matter what:
    1. $|\EmpRisk(s_i) - \EmpRisk(s)| \leq \epsilon$, and
	2. $|\Risk(s_i) - \Risk(s)| \leq \epsilon$
- The $s_i$ **cover** $\ModelClass$, and $q(\epsilon)$ is the **covering number** of $\ModelClass$ (at tolerance $\epsilon$)
    + $q(\epsilon)$ will grow as $\epsilon \rightarrow 0$
    + Think: how many little balls of radius $\epsilon$ would you need to cover some geometric region?
	+ Think of a finite-_length_ interval, say $[0,1]$; how many points $x_i$ in the interval do you need to pick to guarantee that $|x_i-x| \leq \epsilon$? How does that grow with $\epsilon$?  What if it was a 2D area instead of a 1D interval?  etc.
- Consequence:
\begin{eqnarray}
|\EmpRisk(s) - \Risk(s)| & =  & |\EmpRisk(s) - \EmpRisk(s_i) + \EmpRisk(s_i) - \Risk(s_i) + \Risk(s_i) - \Risk(s) |\\
& \leq & |\EmpRisk(s) - \EmpRisk(s_i)|  + |\EmpRisk(s_i) - \Risk(s_i)| +  |\Risk(s) - \Risk(s_i)| \\
& \leq & \epsilon  + |\EmpRisk(s_i) - \Risk(s_i)| + \epsilon\\
& = & 2\epsilon + |\EmpRisk(s_i) - \Risk(s_i)|
\end{eqnarray}

## The route of approximation (2)

- Just saw
\[
|\EmpRisk(s) - \Risk(s)| \leq 2\epsilon + |\EmpRisk(s_i) - \Risk(s_i)|
\]
no matter what
- Now use the union bound: for large $n$, $\max_{i \in 1:q(\epsilon)}{|\EmpRisk(s_i) - \Risk(s_i)|} \leq \epsilon$ with high probability
- So _if_ $q(\epsilon) < \infty$, we can show
\[
\max_{s \in S}{|\EmpRisk(s) - \Risk(s)|} \leq 3\epsilon
\]
with high probability
- $\epsilon$ can be as small as we like so $\Gamma_n \rightarrow 0$


## Pros and cons of the approximation / covering route

#### Pros

- If we can find a covering set, it will often work for _all_ distributions of the data
- We get to do neat math about distances between functions
- Fundamental: if there is uniform convergence, then there (pretty much) _must_ be covering sets with $q(\epsilon) < \infty$

#### Cons

- The union bound is rather loose
    + Deviations aren't mutually exclusive across different $s$'s, there's lots of overlap in those events!
- We need to use sophisticated math about function spaces


## The route of trickery

- Suppose the loss function $\Loss$ is bounded, $0 \leq \Loss \leq m$
- Then $\EmpRisk(s)$ has the [bounded-difference property](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/06/lecture-06.html#(13)); changing _one_ data point can't change $\EmpRisk(s)$ by more than $m/n$
- and $\Gamma_n = \max_{s}{|\EmpRisk(s) - \Risk(s)|}$ [_also_ has the bounded difference property with bound $m/n$](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/06/lecture-06.html#(18))
- [and so](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/06/lecture-06.html#(16))
\[
\Prob{\Gamma_n  \geq \Expect{\Gamma_n} + \epsilon} \leq \myexp{-2n\frac{\epsilon^2}{m^2}}
\]
- $\therefore$ Try to calculate, or bound $\Expect{\Gamma_n}$
- Ideally without knowing any of the true risks...
- Subject for next week


## Summing up

- We've worked on _universal_ convergence of empirical risk to true risks: for each $s \in \ModelClass$,
\[
\Prob{|\EmpRisk(s) - \Risk(s)| \geq \epsilon} \leq \alpha_{s}(n, \epsilon)
\]
- To show that $\EmpRisk(\hat{s}) \rightarrow \Risk(\hat{s})$ and that $\Risk(\hat{s} \rightarrow \Risk(\OptimalModel)$, we want to control $\Gamma_n \equiv \max_{s \in \ModelClass}{|\EmpRisk(s) - \Risk(s)|}$
- **Uniform** convergence means:
\[
\Prob{\Gamma_n \geq \epsilon} \leq \alpha(n, \epsilon)
\]
- If $\ModelClass$ were finite we could use the union bound to get uniform convergence from universal convergence
- If $\ModelClass$ can be well-approximated by a finite number of models, we can use the union bound ("covering number" approach)
    + Theoretically basic but needs too much math
- For bounded losses, $\Gamma_n$ concentrates on $\Expect{\Gamma_n}$, so we'll work on calculating / controlling $\Expect{\Gamma_n}$
    + This will turn out to be much more practical
	
	
