---
title: Bounding the Maximum Deviation with Rademacher Complexity
author: 36-465/665, Spring 2021
date: 2 March 2021 (Lecture 8)
output: slidy_presentation
bibliography: locusts.bib
---


\newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)}
\newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]}
\newcommand{\Risk}{r}
\newcommand{\EmpRisk}{\hat{r}}
\newcommand{\Loss}{\ell}
\newcommand{\OptimalStrategy}{\sigma}
\DeclareMathOperator*{\argmin}{argmin}
\newcommand{\ModelClass}{S}
\newcommand{\OptimalModel}{s^*}
\DeclareMathOperator{\tr}{tr}
\newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}}
\newcommand{\myexp}[1]{\exp{\left( #1 \right)}}
\newcommand{\eqdist}{\stackrel{d}{=}}
\newcommand{\Rademacher}{\mathcal{R}}
\newcommand{\EmpRademacher}{\hat{\Rademacher}}

## Recap

- We control (or at least estimate) $\Risk(\hat{s}) - \EmpRisk(\hat{s})$ and $\Risk(\hat{s}) - \Risk(\OptimalModel)$
- If we can find maximum deviation $\Gamma_n = \max_{s \in \ModelClass}{|\EmpRisk(s) - \Risk(s)|}$ that'd be enough
    + Because [$|\Risk(\hat{s}) - \EmpRisk(\hat{s})| \leq \Gamma_n$](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/07/lecture-07.html#(8)) and [$\Risk(\hat{s}) - \Risk(\OptimalModel) \leq 2\Gamma_n$](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/07/lecture-07.html#(9))
- $\Gamma_n \rightarrow 0$ is [**uniform convergence**](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/07/lecture-07.html#(7)) (of $\EmpRisk(s)$ to $\Risk(s)$)
- Could try to prove uniform convergence by [direct approximation arguments](http://www.stat.cmu.edu/~cshalizi/sml/21/lectures/07/lecture-07.html#(12)), but the math is too hard
- You were promised a short-cut...


## What we really need is the expected maximum deviation {.smaller}

- Remember $|\Risk(\hat{s}) - \EmpRisk(\hat{s})| \leq \Gamma_n$
- Suppose $0 \leq \Loss \leq m$
    + $\Rightarrow$ $\EmpRisk$ has bounded-difference property with bound $m/n$
    + $\Rightarrow$ $\Gamma_n$ also has BDP with bound $m/n$
- Put this all together:
\begin{eqnarray}
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + \epsilon} & \leq & \Prob{\Gamma_n \geq \epsilon}\\
& \leq & \Prob{\Gamma_n - \Expect{\Gamma_n} \geq  \epsilon - \Expect{\Gamma_n}}\\
& \leq & \myexp{-2n(\epsilon-\Expect{\Gamma_n})^2 / m^2}
\end{eqnarray}
- Pick an $\alpha \in (0,1)$, set that equal to the right-hand side, solve for $\epsilon$:
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + \Expect{\Gamma_n} + m\sqrt{\frac{\log{1/\alpha}}{2n}}} \leq \alpha
\]
- Maybe we can calculate/estimate/approximate $\Expect{\Gamma_n}$?




## Maximum deviation vs. expected maximum deviation

- We'll come back to losses and risks specifically later, for now we just have
$Z_1, \ldots Z_n$ all IID, and some class of functions $\mathcal{F}$
    + Abbreviate $Z_1, Z_2, \ldots Z_n$ by $Z_{1:n}$

\[
\Gamma_n = \max_{f \in \mathcal{F}}{\left|\frac{1}{n}\sum_{i=1}^{n}{f(Z_i)} - \Expect{f}\right|}
\]

- $\Gamma_n$ is the **maximum deviation** over $\mathcal{F}$
    + pedantically: maximum absolute deviation of a function from its expected value...
- Just as on last slide, what matters is $\Expect{\Gamma_n} \rightarrow 0$, plus $\Gamma_n$ concentrating around $\Expect{\Gamma_n}$
- Concentrating around the expectation we've got a handle on
- How do we get a handle on $\Expect{\Gamma_n}$, the expected maximum deviation?


## Expected maximum deviation vs. expected maximum discrepancy {.smaller}

- Pretend we have a **ghost sample** $Z^{\prime}_1, \ldots Z^{\prime}_n$ which have the same distribution as $Z_i$ but are independent
\begin{eqnarray}
\left| \left(\frac{1}{n}\sum_{i=1}^{n}{f(Z_i)}\right) - \Expect{f} \right| & = & \left| \frac{1}{n}\sum_{i=1}^{n}{(f(Z_i) - \Expect{f})}\right|\\
& = & \left|\frac{1}{n}\sum_{i=1}^{n}{f(Z_i) - \Expect{f(Z^{\prime}_i)}}\right|\\
& = & \left| \frac{1}{n}\sum_{i=1}^{n}{\Expect{f(Z_i) - f(Z^{\prime}_i)|Z_i}} \right| \\
& = & \left| \Expect{\frac{1}{n}\sum_{i=1}^{n}{f(Z_i) - f(Z^{\prime}_i)|Z_{1:n}}} \right|
\end{eqnarray}
because all the $Z^{\prime}_i$s are independent of the $Z_i$s
- $n^{-1} \sum_{i=1}^{n}{f(Z_i) - f(Z^{\prime}_i)}$ is the **discrepancy** between two sample averages for $f$
    + $=$ how much the sample average changes just due to sampling noise
	

##	 Expected maximum deviation vs. expected maximum discrepancy (2)
	
\begin{eqnarray}
\Gamma_n = \max_{f \in \mathcal{F}}{\left| \left( \frac{1}{n}\sum_{i=1}^{n}{f(Z_i)} \right) - \Expect{f} \right| } 
& = & \max_{f \in \mathcal{F}}{ \left| \Expect{\frac{1}{n}\sum_{i=1}^{n}{f(Z_i) - f(Z^{\prime}_i)} \mid Z_{1:n} } \right| }\\
\end{eqnarray}
- But $\max{\left| \Expect{\cdot} \right| } \leq \Expect{\max{\left| \cdot \right|}}$ so
\begin{eqnarray}
\Gamma_n & \leq & \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{f(Z_i) - f(Z_i^{\prime})} \right|} \mid Z_{1:n}}\\
\Expect{\Gamma_n} & \leq & \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{f(Z_i) - f(Z_i^{\prime})} \right|}}
\end{eqnarray}
- The expected maximum _deviation_ $\leq$ the expected maximum _discrepancy_



## Expected maximum discrepancy to Rademacher complexity {.smaller}

- We want to understand
\[
\Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{f(Z_i) - f(Z_i^{\prime})} \right|}}
\]
- Focus on $n^{-1} \sum_{i=1}^{n}{f(Z_i) - f(Z_i^{\prime})}$
- Because $Z^{\prime}_i \eqdist Z_i$ but they're independent so $f(Z_i) - f(Z_i^{\prime}) \eqdist f(Z_i^{\prime}) - f(Z_i)$
- Introduce new random variables $\sigma_i$, $=\pm 1$ with equal probability, independent of each other and of $Z$ and $Z^{\prime}$
    + **Rademacher** random variables (but $\sigma$ for "s" for "sign")
\begin{eqnarray}
\sigma_i (f(Z_i) - f(Z_i^{\prime})) & \eqdist & f(Z_i) - f(Z_i^{\prime})\\
\frac{1}{n}\sum_{i=1}^{n}{f(Z_i) - f(Z_i^{\prime})} & \eqdist & \frac{1}{n}\sum_{i=1}^{n}{\sigma_i (f(Z_i) - f(Z_i^{\prime}))}\\
\Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{f(Z_i) - f(Z_i^{\prime})} \right|}} & = & \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i (f(Z_i) - f(Z_i^{\prime}))} \right|}}\\
& \leq & \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|}} + \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{-\sigma_i f(Z^{\prime}_i)}\right|}}\\
& = & 2\Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|}}\\
& \equiv & 2 \Rademacher_n(\mathcal{F})
\end{eqnarray}



## Rademacher complexity {.smaller}

- The **Rademacher complexity** of a function class $\mathcal{F}$, on $n$ samples, is
\[
\Rademacher_n(\mathcal{F}) \equiv \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|}}
\]
    + Often omit the $\mathcal{F}$ when clear from context
- Some people include a factor of 2 in the definition because, as we just saw,
\[
\Expect{\Gamma_n} \leq 2 \Rademacher_n
\]
- Unpack $\Rademacher_n$ from the inside out:
\[
\left|\frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right| = \text{magnitude of sample covariance between function $f$ and binary noise $\sigma$}
\]
\[
\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|} = \text{maximum apparent covariance between our functions and binary noise}
\]
\[
 \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|}} = \text{expected maximum apparent covariance between our functions and noise}
\]


## Rademacher complexity (2)

- $\Rademacher_n(\mathcal{F}) =$ how well can functions from $\mathcal{F}$ correlate with pure binary noise?
- High Rademacher complexity $=$ $\mathcal{F}$ seems like it can fit anything (including video snow)
- Low Rademacher complexity $=$ $\mathcal{F}$ just can't fit pure noise well
- Aside: Humans (or at least undergrads at my alma mater) are [remarkably good at coming up with stories that seem to fit with random noise](http://bactra.org/weblog/926.html), and their learning performance is predicted by their Rademacher complexities [@human-rademacher-complexity]


## Rademacher complexity and generalization error

- Remember from above: for any $\alpha \in (0,1)$,
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + \Expect{\Gamma_n} + m\sqrt{\frac{\log{1/\alpha}}{2n}}} \leq \alpha
\]
- But $\Expect{\Gamma_n} \leq 2\Rademacher_n$ so
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + 2\Rademacher_n + m\sqrt{\frac{\log{1/\alpha}}{2n}}} \leq \alpha
\]

- This is our first **generalization error bound**
   + Remember this assumes a bounded loss function, $0 \leq \Loss \leq m$


## How do we calculate the Rademacher complexity? {.smaller}

\[
\Rademacher_n(\mathcal{F}) = \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|}}
\]

- Could try to do math...
    + Lots of clever results on $\Rademacher(\mathcal{F})$ for various $\mathcal{F}$
    + Some useful references for such results: @Bartlett-Mendelson-on-Rademacher-complexity, @Mohri-Rostamizadeh-Talwalkar (see under particular model classes)
	
## How do we calculate the Rademacher complexity? {.smaller}

\[
\Rademacher_n(\mathcal{F}) = \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|}}
\]


- We could do statistics: fix (condition on) $Z_1, \ldots Z_n$, and average over $\sigma$:
\begin{eqnarray}
\EmpRademacher_n(\mathcal{F}) & \equiv &  \Expect{\max_{f \in \mathcal{F}}{\left| \frac{1}{n}\sum_{i=1}^{n}{\sigma_i f(Z_i)}\right|}|Z_{1:n}}\\
\Rademacher_n(\mathcal{F}) & = & \Expect{\EmpRademacher_n(\mathcal{F}) }
\end{eqnarray}
- $\EmpRademacher_n(\mathcal{F})$ is the **empirical Rademacher complexity** of $\mathcal{F}$, and it is random (because it's a function of $Z_{1:n}$)
- Offline: $\EmpRademacher_n$ again has the bounded difference property, with bound $m/n$ for each $Z_i$
- $\therefore$ $\Prob{\Rademacher_n \geq \EmpRademacher_n + \epsilon} \leq \myexp{-2n\epsilon^2/m^2}$ and we can use the empirical Rademacher complexity in our bounds (at some cost in confidence / margin of error)
\[
\Prob{\Risk(\hat{s}) \geq \EmpRisk(\hat{s}) + 2\EmpRademacher_n + 3m\sqrt{\frac{\log{2/\alpha}}{2n}}} \leq \alpha
\]


## Even more approximation

- Simulate a $\sigma_i$ sequence and do the optimization $\max_{f \in \mathcal{F}}{| n^{-1}\sum_{i=1}^{n}{\sigma_i f(z_i)} |}$
    + i.e., hold the $Z$s fixed at their observed values, then independently make up $\sigma$s, then search for the function which maximizes the sample covariance between $\sigma_i$ and $f(z_i)$
- This is an unbiased estimate of $\EmpRademacher$
- Again, the bounded-difference property applies, so we can use the average of such simulations to estimate $\EmpRademacher$
- $\EmpRademacher$ estimates $\Rademacher$
- $2\Rademacher$ upper-bounds $\Expect{\Gamma}$
- which gives us a bound on $\Risk(\hat{s}) - \EmpRisk(\hat{s})$
- Some book-keeping but we'll do that in the homework!



## Summing up

- We want to know the expected maximum deviation $\Expect{\Gamma_n}$ of the empirical risk from the true risk
- $\Expect{\Gamma_n} \leq$ the expected maximum discrepancy between empirical risks on two independent samples
- The expected maximum discrepancy is $\leq 2\times$ expected maximum covariance with binary noise
- $\therefore$ $\Expect{\Gamma_n} \leq 2\Rademacher_n$
- Sometimes people can calculate $\Rademacher_n$
- Otherwise we approximate it on the training data (empirical Rademacher complexity, simulations)




## Backup: Strategies vs. Losses

- Set $Z_i = (X_i, Y_i)$
- Set $f(Z_i) = \Loss(Y_i, s(X_i))$
- Each strategy $s \in S$ corresponds to an $f \in \mathcal{F}$
    + $\mathcal{F}$ is called the **loss class**; people sometimes even call an $f$ a **loss function**, but that's really confusing so please don't
- The Rademacher complexity we really care about is the R.C. of $\mathcal{F}$
- Often it's easier to get the Rademacher complexity of $S$, especially when "actions" take numerical values
- Useful definition: a function $g$ is **Lipsschitz with constant $h$**, or **$h$-Lipschitz**, if $|g(x) - g(y)| \leq h d(x,y)$, where $d(x,y)$ is the distance (in whatever metric makes sense) between $x$ and $y$
    + If $g$ is differentiable, $h$ is the maximum (absolute) derivative
	+ Examples: $|x|$ is Lipschitz; $\sin{x}$ is Lipschitz; $x^2$ is Lipschitz on the interval $[-m, m]$ (can you find the constants in each case?)
    + Non-examples: $\Indicator{x \leq a}$ is not Lipschitz; $1/x$ is not Lipschitz on $(0, a)$; $\log{p}$ is not Lipschitz on $(0,1)$
- Useful fact (which we will not prove): if $\Loss(y, y^{\prime})$ is $h$-Lipschitz in $y-y^{\prime}$, with $\Loss(y,y)=0$, then $\Rademacher_n(\mathcal{F}) \leq 2h \Rademacher(S)$ [@Bartlett-Mendelson-on-Rademacher-complexity, theorem 12]
- Why it's a useful fact: it lets us calculate the Rademacher (or empirical Rademacher) complexity of the strategies and then multiply to get the Rademacher complexity we need for generalization error bounds


## References

