---
title: "Deivation Inequalities II: From the Chernoff Bounds to the Bounded-Difference Inequality"
author: 36-465/665, Spring 2021
date: 18 February 2021 (Lecture 6)
output: slidy_presentation
---

\[
\newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)}
\newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]}
\newcommand{\Risk}{r}
\newcommand{\EmpRisk}{\hat{r}}
\newcommand{\Loss}{\ell}
\newcommand{\OptimalStrategy}{\sigma}
\DeclareMathOperator*{\argmin}{argmin}
\newcommand{\ModelClass}{S}
\newcommand{\OptimalModel}{s^*}
\DeclareMathOperator{\tr}{tr}
\newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}}
\newcommand{\myexp}[1]{\exp{\left( #1 \right)}}
\]



## Last time

- We want to give high-probability bounds on how far $Z$ is from $\Expect{Z}$
    + Ultimately we want to show $\Prob{|\EmpRisk(\hat{s}) - \Risk(\hat{s})| \geq g(n,\alpha)} \leq \alpha$, or $\Prob{|\Risk(\hat{s}) - \Risk(s^*)| \geq h(n,\alpha)} \leq \alpha$ but we've got to build up to that
- Markov's inequality: $\Prob{Z \geq \epsilon} \leq \Expect{Z}/\epsilon$
- Markov $\Rightarrow$ Chebyshev: $\Prob{|Z-\Expect{Z}| \geq \epsilon} \leq \Var{Z}/\epsilon^2$
-  $\Rightarrow$ If $X_i$ are IID then $\Prob{|\overline{X}_n - \Expect{X}|\geq \epsilon} \leq \Var{X}/n\epsilon^2$
- Exponential Markov/Chernoff bounds: $\Prob{Z  \geq \epsilon} \leq \min_{t > 0}{e^{-t\epsilon} \Expect{e^{tZ}}}$ and $\Prob{Z \leq \epsilon} \leq \min_{t > 0}{e^{t\epsilon} \Expect{e^{-tZ}}}$
- $Z$ is sub-Gaussian $\Leftrightarrow$ $\Expect{e^{tZ}} \leq e^{t\Expect{Z}} e^{t^2 c^2/2}$ for some $c>0$
- If $X_i$ are IID and sub-Gaussian, then $\Prob{|\overline{X}_n - \Expect{X}|\geq \epsilon} \leq 2\myexp{-n\epsilon^2/2c^2}$


## This time

- Deviation inequalities for sums/averages of independent but not identically distributed (INID) variables
- Deviation inequalities for functions of multiple random variables which are not sums or averages


## Sums of INID variables {.smaller}

- If $X_1$ and $X_2$ are independent, then $\Expect{f(X_1) g(X_2)} = \Expect{f(X_1)}\Expect{g(X_2)}$ for any functions $f, g$
- $\therefore$ $\Expect{e^{t(X_1+X_2)}} =\Expect{e^{tX_1} e^{tX_2}} = \Expect{e^{t X_1}} \Expect{e^{t X_2}}$ if $X_1$ and $X_2$ are independent
- $\therefore$ $M_{X_1+X_2}(t) = M_{X_1}(t) M_{X_2}(t)$ if $X_1$ and $X_2$ are independent
- If $X_1$ and $X_2$ are independent sub-Gaussians with constants $c_1^2$, $c_2^2$,
then $X_1+X_2$ is sub-Gaussian with constant $c_1^2+c_2^2$
\[
M_{X_1+X_2}(t) = M_{X_1}(t) M_{X_2}(t) \leq e^{t\Expect{X_1}} e^{t\Expect{X_2}} e^{t^2 c_1^2/2} e^{t^2 c_2^2/2}
\]
- If $X_i$ are all sub-Gaussian with constants $c_i^2$ and expectations $\mu_i$, and they're all independent, then
\[
\Prob{\left|\sum_{i=1}^{n}{X_i} -\sum_{i=1}^{n}{\mu_i}\right|\geq \epsilon} \leq 2\myexp{-\frac{\epsilon^2}{2\sum_{i=1}^{n}{c_i^2}}}
\]
- Off-line: 
    + What's the deviation bound for the average (not sum) of the $X_i$?
	+ How do the deviation bounds for the sum and average simplify if all the $c_i^2 \leq C^2$ for some $C>0$?


## Hoeffding's inequality

- Suppose $Z \in [a, b]$ with probability 1, with expectation $\mu$

> **Hoeffding's bound**: $M_{Z}(t) \leq e^{t\mu} e^{t^2(b-a)^2/8}$

- Proof: _basically_ that $\Var{Z} \leq (b-a)^2/4$, plus some extra probability to argue that this upper-bounds the MGF
    + Details not very important, in online handout to be posted on class website
- Implication: a bounded random variable is sub-Gaussian, with $c^2 = (b-a)^2/4$

## Hoefdding's inequality (2)

> - Suppose $Z_i \in [a_i, b_i]$, with expectation $\mu_i$
> - For each $Z_i$, $M_{Z_i}(t) \leq e^{t\mu_i} e^{t^2(b_i - a_i)^2/8}$
>     + Proof: previous slide
> - **Hoeffding's bound** or **Hoeffding's inequality**:
\[
\Prob{\left|\sum_{i=1}^{n}{Z_i - \mu_i}\right| \geq \epsilon} \leq 2 \myexp{-\frac{\epsilon^2}{2\sum_{i=1}^{n}{(b_i - a_i)^2/4}}}
\]
> - Proof: Multiply $M_{Z_i}$ together to get the MGF of the sum, then apply the sub-Gaussian bound
>     + i.e., combine the previous two slides


## Hoeffding's inequality (3)

> - Suppose $Z_i \in [a_i, b_i]$, with expectation $\mu_i$
> - Define $\overline{Z}_n \equiv n^{-1}\sum_{i=1}^{n}{Z_i}$
> - Define $\overline{\mu}_n \equiv n^{-1}\sum_{i=1}^{n}{\mu_i}$
- **Hoeffding's inequality** or **Hoeffding's bound**:
\[
\Prob{\left|\overline{Z}_n - \overline{\mu}_n\right| \geq \epsilon} \leq 2\myexp{-\frac{n^2\epsilon^2}{2\sum_{i=1}^{n}{(b_i - a_i)^2/4}}}
\]
>    + Proof: if the average is $\epsilon$ away from its expectation, the sum must be $n\epsilon$ away from _its_ expectation; apply previous slide
> - Numerator in the exponent is $\propto n^2$, denominator has $n$ terms, _typically_ we get an $O(n)$ rate


## Example: log loss

- Say we're doing classification, so we're predicting $Y=0$ or $Y=1$, based on $X$, we _think_ $\Prob{Y=1|X=x} = q(x)$ but really $\Prob{Y=1|X=x} = p(x)$
- $Z_i =$ the log loss we experience on data point $(X_i, Y_i)$
\begin{eqnarray}
Z_i & = & -Y_i \log{q(X_i)} - (1-Y_i) \log{(1-q(X_i))}\\
& = & -\log{(1-q(X_i))} - Y_i \log{\frac{q(X_i)}{1-q(X_i)}}\\
\Expect{Z_i|X=x} & = & -\log{(1-q(x))} - p(x) \log{\frac{q(x)}{1-q(x)}}
\end{eqnarray}
    + Note how the log odds ratio shows up
        * This is why logistic regression is such a big deal for classification...
		
## Example: log loss (2) {.smaller}

- Given $X_i=x_i$, each $Z_i$ can take only 1 of 2 values
- $\therefore$ for each $Z_i$ we have $|b_i - a_i| = \left|\log{\frac{q(x_i)}{1-q(x_i)}}\right|$
- Assume $|\log{\frac{q(x_i)}{1-q(x_i)}}| \leq w$
    + "logit-bounded" or "bounded log-odds ratio"
- $\therefore$
\[
\Prob{\left|\overline{Z}_n - \overline{\mu}_n\right| \geq \epsilon \mid| X_1,\ldots X_n} \leq 2\myexp{-\frac{n^2 \epsilon^2}{\frac{1}{2}n w^2}}
\]
- Notice $X_i$ don't appear in the bound, so
\[
\Prob{\left|\overline{Z}_n - \overline{\mu}_n\right| \geq \epsilon}
\leq 2\myexp{-2\frac{n \epsilon^2}{w^2}}
\]
- Conclusion: if we use the log loss for classification, and the log odds ratio is bounded, the empirical risk is exponentially unlikely to be far from the true risk
    + ... for a particular fixed model
         * We'll see in later classes how to make this bound apply to _every_ model in some big class at once
    + This result is also useful in many other situations with binary data, not just classification (de-noising binary signals or images, social network analysis, ...)


## What about functions that aren't sums or averages?

- One idea: Taylor series / linearization:
    + if each $X_i \approx \mu_i + \Delta_i$
    + and the $\Delta_i$ are all small,
	+ and $\partial f/\partial x_i$ is small
	    * or, really, if the product $\Delta_i \partial f/\partial x_i$ is small
	+ then $f(X_1, \ldots X_n) \approx f(\mu_1, \ldots \mu_n) + \sum_{i=1}^{n}{\Delta_i (\partial f/\partial x_i)}$
	+ This is a sum and we know how to give deviation bounds for it
- Drawback: we're back to asymptotic approximations again
- Idea we can take away from this: if every input to the function can only change it a little, and the inputs are independent, then there should be a deviation inequality


## The Efron-Stein Inequality

> Suppose $X_1, \ldots X_n$ are independent but not necessarily identically distributed, and $Z=f(X_1, \ldots X_n)$.  Say $Y_1, \ldots Y_n$ are independent copies of the $X_i$, and define $Z_i=f(X_1, \ldots X_{i-1}, Y_i, X_{i+1}, \ldots X_n)$.  Then
\[
\Var{Z} \leq \frac{1}{2}\sum_{i=1}^{n}{\Expect{(Z-Z_i)^2}}
\]

- _Proof_: a bit complicated and uses a special probability technique ("martingales"), so see notes that'll be on the class website
    + A key step is more intuitive: $\Var{X} = \frac{1}{2}\Expect{(X-Y)^2}$ where $Y$ is an independent copy of $X$
- _Moral_: randomly changing one input at a time doesn't change the function too much $\Rightarrow$ randomly changing all the inputs independent can't change the function too much


## The bounded difference property

- A function $f(x_1, \ldots x_n)$ has the **bounded difference property** when
changing just one input can only make a limited difference to the function:
\[
|f(x_1, \ldots, x_{i-1}, x_i, x_{i+1}, \ldots x_n) - f(x_1, \ldots x_{i-1}, y_i, x_{i+1}, \ldots x_n)| \leq d_i
\]
- Some simple examples:
    + $\sin{\left(\sum_{i=1}^{n}{w_i x_i}\right)}$, with $d_i=2$ for all $i$
	+ $\mathrm{logit}^{-1}\left(\sum_{i=1}^{n}{\beta_i x_i}\right)$ with $d_i=1$ for all $i$
    + We will see more examples later


## Bounded difference + Efron-Stein

- Remember we're writing $Z=f(X_1, \ldots X_n)$, and $Z_i = f(X_1, \ldots Y_i, \ldots X_n)$ where $Y_i$ has the same distribution as $X_i$ but is independent
- Bounded difference property $\Rightarrow$ $\Expect{(Z-Z_i)^2} \leq d_i^2$
- Thus $\Var{Z} \leq \frac{1}{2}\sum_{i=1}^{n}{d_i^2}$
- Now use Chebyshev:
\[
\Prob{|Z-\Expect{Z}|\geq\epsilon} \leq \frac{1}{2}\frac{\sum_{i=1}^{n}{d_i^2}}{\epsilon^2}
\]
- Can we do better if we assume more?


## Hoeffding + bounded differences = McDiarmid

The **bounded difference** or **McDiarmid** inequality

> If $f$ has the bounded difference property, then
\[
\Prob{|Z-\Expect{Z}|\geq\epsilon} \leq 2\myexp{-\frac{2\epsilon^2}{\sum_{i=1}^{n}{d_i^2}}}
\]

- _Proof_: Again, see handout online if you're interested
- _Moral_: If changing each input can only alter the function by so much, and the inputs are independent, then probability of large deviations is exponentially small


## Making the bounded-differences inequality useful

- Here's the bounded-differences inequality again
\[
\Prob{|Z-\Expect{Z}|\geq\epsilon} \leq 2\myexp{-\frac{2\epsilon^2}{\sum_{i=1}^{n}{d_i^2}}}
\]
    + Remember $Z=f(X_1, \ldots X_n)$ and changing $X_i$ (alone) alters $Z$ by at most $d_i$
- If $\sum_{i=1}^{n}{d_i^2} = O(n)$ this isn't helpful because the bound gets _weaker_ as $n$ grows
    + Example: $f(X_1, \ldots X_n) = \sum_{i=1}^{n}{X_i}$ with the $X_i$'s bounded (as in the sum form of Hoeffding's bound)
    + Example: $\sin{\left(\sum_{i=1}^{n}{\beta_i X_i}\right)}$  with the $X_i$'s whatever they want to be (but $\beta_i$ fixed as $n$ changes)
- If the $d_i = O(1/n)$ then $\sum_{i}{d_i^2} = nO(1/n^2) = O(1/n)$ and this is useful
     + Example: averaging bounded random variables (as in the mean form of Hoeffding)
	 
 
## Example: Empirical vs. true distribution

- Say the $X_i$ are IID on the real line, with true CDF $F^*$
- Define the **empirical CDF** as
\[
\hat{F}_n(a) = \frac{1}{n}\sum_{i=1}^{n}{\Indicator{X_i \leq a}}
\]
- Note: $\Expect{\hat{F}_n(a)} = F^*(a)$ (for all $a$, $n$)
- Bounded difference property: for each $i$, $a$, changing $X_i$ can change $\hat{F}_n(a)$ by at most $1/n$, so $d_i=1/n$
- Implication: For any $a \in \mathbb{R}$,
\[
\Prob{\left|\hat{F}_n(a) - F^*(a)\right| \geq \epsilon} \leq 2\myexp{-2n\epsilon^2}
\]

## Example: Empirical vs. true distribution (2)

- Define the maximum deviation as
\[
M_n \equiv \max_{a}{\left|\hat{F}_n(a) - F^*(a)\right|}
\]
- Bounded difference property: Changing any one $X_i$ can change $M_n$ by at most $1/n$, so $d_i=1/n$ again
    + Because that's the most it can change any of the $\hat{F}_n(a)$
- So
\[
\Prob{\left|M_n - \Expect{M_n}\right| \geq \epsilon} \leq 2 \myexp{-2n\epsilon^2}
\]
- If we can show that $\Expect{M_n} \rightarrow 0$, then we'll have shown the whole function $\hat{F}_n \rightarrow F^*$
    + This is called "uniform convergence"

 

## Summing up

> - Moment-generating functions multiply when we add independent variables $\Rightarrow$ we can handle sums and averages that aren't IID
>    + Especially if everything stays sub-Gaussian
> - Hoeffding: bounded random variables are sub-Gaussian
> - Bounded difference property: changing one input to a function can't change the output by more than a certiain amount
> - BDP $\Rightarrow$ bounded variance (Efron-Stein)
> - BDP $\Rightarrow$ sub-Gaussian-style deviation inequality (bounded difference or McDiarmid inequality)

## Going forward:

>- Can we get more results that will hold for many different functions?
>    + Think of the bounded-difference inequality, but weaker or just different properties on the functions
>    + **Concentration inequalities**; next week
> - Can we get results that hold for _infinitely many_ functions at once?
>    + Like the bound on $M_n$ two slides ago
>    + **Uniform convergence** results; week after next
> - Can we _please_ get back to learning prediction strategies?
>    + Immediately after that, I promise!

	
