---
title: Details and Complements for Lecture 6
author: 36-465/665, Spring 2021
date: 18 February 2021
bibliography: locusts.bib
output: bookdown::pdf_document2
---

\newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)}
\newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]}
\newcommand{\Var}[1]{\mathrm{Var}\left[ #1 \right]}
\newcommand{\Cov}[1]{\mathrm{Cov}\left[ #1 \right]}
\newcommand{\Risk}{r}
\newcommand{\EmpRisk}{\hat{r}}
\newcommand{\Loss}{\ell}
\newcommand{\OptimalStrategy}{\sigma}
\newcommand{\ModelClass}{S}
\newcommand{\OptimalModel}{s^*}
\newcommand{\Indicator}[1]{\mathbb{1}\left\{ #1 \right\}}
\newcommand{\myexp}[1]{\exp{\left( #1 \right)}}


# Notation

- Upper case = random variable, lower case = realization
- When $X_1, \ldots X_n, \ldots$ is a sequence of random variables, $X_{1:n}$ abbreviates the finite sequence $X_1, X_2, \ldots X_{n-1}, X_n$, and $X$ indicates the entire infinite random sequence.  This may suggest that $X_n$ is just a coordinate of $X$; it should.


# Bound on the variance of a bounded-range random variable

```{proposition}
Suppose $Z$ is a scalar random variable confined to some interval, so that  $\Prob{a \leq Z \leq b} = 1$.  Then $\Var{Z} \leq (b-a)^2/4$.
```

```{proof}
First, we will maximize the variance of bounded-range random
variables which are supported only at the end-points of the range.  Then we
will argue that shifting probability into the interior of the range can only
reduce the variance, so the apparently-special case we considered first is all
we really need.

So, consider random variables where $Z=b$ with probability $p$, and $Z=a$ with
probability $1-p$.  The expectation is clearly
\begin{equation}
\Expect{Z} = bp + (1-p)a = a + (b-a)p
\end{equation}
and the variance is
\begin{eqnarray}
\Expect{(Z-\Expect{Z})^2} & = & p(b-[a+(b-a)p])^2 + (1-p)(a-[a+(b-a)p])^2\\
& = & p (b-a-(b-a)p)^2 + (1-p)(a-a-(b-a)p)^2\\
& = & p ((b-a)(1-p))^2 + (1-p)((b-a)p)^2\\
& = & p (b-a)^2 (1-p)^2 + (1-p)(b-a)^2 p^2\\
& = & (b-a)^2 p(1-p)[(1-p) + p]\\
& = & (b-a)^2 p(1-p)
\end{eqnarray}
This is clearly maximized when $p=1/2$, yielding a maximum value for $\Var{Z}$ of $(b-a)^2/4$.

The above result was for distributions which divide their probability between
$Z=a$ and $Z=b$; let us call these the "boundary distributions".  Fix any one
such distribution, with mean $\mu = a+(b-a)p$.  Now consider any other,
non-boundary distribution on $[a,b]$ with the same expectation $\mu$.  Clearly,
this distribution must put more probability mass closer to $\mu$ than the
boundary distribution we started with.  But since $\Var{Z} =
\Expect{(Z-\Expect{Z})^2}$, this distribution must have a smaller variance than
the two-point distribution with the same $\mu$.  Moreover, every non-boundary
distribution has the same expectation as some boundary distribution.  (This is
because every distribution on a finite interval $[a,b]$ (i) has a well-defined
expected value, and (ii) that expected value is also in the interval $[a,b]$,
which (iii) means it equals $a+(b-a)p$ for some $p\in[0,1]$.)  So every
non-boundary distribution has less variance than the boundary distribution with
the same expectation.
Hence the distribution of maximum variance must be a boundary
distribution.  But we have just found the distribution which maximizes variance within the boundary distributions.
```


# Hoeffding's bound on the moment generating function

```{proposition}
Suppose $Z$ is confined to the interval $[a,b]$, with expected value $\mu$.  The  $M_{Z}(t) \leq e^{t\mu} e^{t^2 (b-a)^2/8}$, i.e., $Z$ is sub-Gaussian with scale of at most $(b-a)^2/4$.
```

```{proof}
The factor of $e^{t\mu}$ in the result just arises from "pulling
out" the expectation of $Z$, so the real assertion is that $M_{Z-\Expect{Z}}(t)
\leq e^{t^2 (b-a)^2/8}$.  One might think that this would require calculations
to bound the third, fourth, etc., central moments, rather in the fashion that
our previous result bounded the variance.  However, a somewhat indirect
approach, that goes back to Hoeffding, avoids this at the cost of look a bit
strange at first.

First, let's define $C \equiv Z-\Expect{Z}$, and $K(t) \equiv \log{M_C(t)}$.
The result we care about is equivalent to the statement that $K(t) \leq t^2
(b-a)^2/8$, so if we can establish the latter we've won.

Hoeffding's trick for doing so is to consider a _broader_ family of
distributions than just that of $C$.  Say that $C$ has pdf[^pdf] $p(z)$, and,
for each $\theta$, define $C_{\theta}$ as the random variable with pdf
\begin{equation}
p_{\theta}(z) = p(z) \myexp{\theta z - K(\theta)}
\end{equation}
Notice that $p_{\theta}(z) \geq 0$ and that $\int{p_{\theta}(z) dz} = \int{p(z)
e^{\theta z} dz} e^{-K(\theta)} = \frac{M_C(\theta)}{M_C(\theta)} = 1$, so
$p_{\theta}(z)$ is a valid probability density function for each $\theta$.

[^pdf]: If you're worried about non-continuous distributions, you have a point (pardon the pun), but this can be handled by using measure-theoretic probability.  Let $P$ be the probability measure of $C$, and say that $P_{\theta}$ is the measure absolutely continuous with respect to $P$ with density (=Radon-Nikodym derivative) $\frac{dP_{\theta}}{dP} = \myexp{\theta z - K(\theta)}$.  The arguments in the text then go through unchanged.

This is an example of what's called an "exponential family" distribution, with
"base distribution" $p(z)$.  One of the basic properties of such a distribution
is that we can recover its moments from the function $K(\theta)$.  Thus
\begin{eqnarray}
\frac{dK}{d\theta} & = & \frac{d}{d\theta} \log{\Expect{e^{\theta C}}} \\
& = & \frac{1}{\Expect{e^{\theta C}}}\Expect{\frac{d}{d\theta} e^{\theta C}}\\
& = & e^{-K(\theta)}\Expect{ C e^{\theta C}}\\
& = & \Expect{C e^{\theta C - K(\theta)}}
\end{eqnarray}
which is $\Expect{C_{\theta}}$

Similarly,
\begin{eqnarray}
\frac{d^2 K}{d\theta^2} & = & \Var{C_{\theta}}\\
& \leq & \frac{(b-a)^2}{4}
\end{eqnarray}
using the result of the previous section.

Finally, let us Taylor expand $K(t)$ around 0.  Taylor's theorem in this case becomes exact, not an approximation, for some point between the expansion point and $t$, substituted into the last derivative, thus:
\begin{equation}
K(t) = K(0) + t \frac{dK}{d\theta}(0) + \frac{t^2}{2}\frac{d^2K}{d\theta^2}(\theta)
\end{equation}
where $0 \leq \theta \leq t$.  But $K(0)=0$, $\frac{dK}{d\theta}(0) = \Expect{C} = 0$, and $\frac{d^2K}{d\theta^2}(\theta) \leq (b-a)^2/4$, so
\begin{equation}
K(t) \leq \frac{t^2}{2}\frac{(b-a)^2}{4} = t^2 \frac{(b-a)^2}{8}
\end{equation}
```


# Martingales and Martingale Differences

I don't know of any way of proving results like the Efron-Stein inequality or
the bounded difference / McDiarmid inequality without using ideas from
probability theory about types of random sequences called "martingales" and
"martingale difference sequences".


A sequence of random variables $Z$ is a **martingale**[^martingale] when the conditional expectation of the next value is the present value:
\begin{equation}
\Expect{Z_{n+1}|Z_1, Z_2, \ldots Z_n} = Z_n
\end{equation}
By extension, the sequence $Z$ is a martingale **with respect to the sequence $X$** when

- $Z_n$ is a function[^measurable] of $X_{1:n}$, and
- the conditional expectation of the next value given the history is the current value:
\begin{equation}
\Expect{Z_{n+1}|X_1, \ldots X_n} = Z_n
\end{equation}
A martingale, in the simple sense, is a martingale with respect to itself.

[^martingale]:In English, the word "martingale" original meant a set of straps
attached to a horse's head to keep the animal from raising its head too high.
The word came to English from French, which got it from Spanish, which got it
from the Arabic word _al marta'a_, "the fastening".  It then somehow became the
name of a gambling scheme where you keep doubling your bet every time you lose,
in the hope of coming out ahead in one throw.  (Maybe with the idea that this
rule was restraining how high you bet?) In the 1930s, French mathematicians
trying to define the notion of probability in terms of betting needed a name
for processes that represented the stake of a gambler in a fair game, and
somehow seized on "martingale".  Random processes whose next increment averages
to zero have since turned out to be extremely useful technical tools in
probability theory, even in contexts that have nothing to do with gambling.

[^measurable]: Strictly speaking, $Z_n$ must be a "measurable" function of $X_{1:n}$.  A very rough explanation of "measurable function" would go as follows.  Suppose we know that the argument $X$ to a function $f$ is in some set $A$, in stmbols $X \in A$ for some set $A$.  Then we know that $f(X) \in f(A)$, the "image" of $A$ under the function $f$.  Now consider what happens if we know that $X$ is in a sequence of nested, increasingly small sets $A_n$, which narrow down to a particular point $x$.  The function $f$ is "measurable" if the image sets $f(A_n)$ also narrow down to the single point $f(x)$.  This is a somewhat weaker requirement that $f$ being continuous.  (For instance, the indicator function for any interval on the real line is measurable, but discontinuous.)  The point of requiring a measurable function, rather than just any function, is to rule out very erratic, "pathological" functions which do not play nicely with probability distributions.  --- If you know enough measure theory, or measure-theoretic probability, to poke holes in this explanation, you also know the real definition of "measurable function" as well as I do.  If this is your first exposure to such concepts and are curious to learn more, I recommend either @Grimmett-Stirzaker, or @Pollard-users-guide.  (The classic book by @Halmos-measure-theory is great but a bit abstract.)

```{theorem}
If $Z$ is a martingale with respect to $X$, then for any $m > n$,
\begin{equation}
\Expect{Z_m|X_{1:n}} = Z_n
\end{equation}
```

```{proof}
An equivalent statement is that for any $h>0$, $\Expect{Z_{n+h}|X_{1:t}} = Z_n$.  We will prove this by mathematical induction, since it's clearly
true for $h=1$ (by the definition of a martingale).
\begin{eqnarray}
\Expect{Z_{n+h+1}|X_{1:t}} & = & \Expect{\Expect{Z_{n+h+1}|X_{1:n+h}}|X_{1:t}}\\
& = & \Expect{Z_{n+h}|X_{1:t}}\\
& = & Z_n
\end{eqnarray}
using the martingale property in the middle line, and the inductive hypothesis in the last line.
```



The random sequence $D_n$ is a **martingale difference sequence** with respect to $X$ when
- $D_n$ is a function of $X_1, \ldots X_n$, and
- the conditional expectation of the next value give the past is always zero:
\begin{equation}
\Expect{D_{n+1}|X_1, \ldots X_n} = 0
\end{equation}

Our next result justifies the name "martingale difference sequence":

```{proposition}
If $Z$ is a martingale with respect to $X$, and we define $D_n \equiv Z_n - Z_{n-1}$, then $D$ is a martingale difference sequence with respect to $X$.
```

```{proof}
\begin{eqnarray}
\Expect{D_{n+1}|X_{1:n}} & = & \Expect{Z_{n+1}-Z_n|X_{1:n}}\\
& = & \Expect{Z_{n+1}|X_{1:n}} - \Expect{Z_n|X_{1:n}}\\
& = & Z_n - Z_n = 0
\end{eqnarray}
where in the last step we use the facts that $Z_n$ is a function of $X_{1:n}$, and that $\Expect{Z_{n+1}|X_{1:n}} = Z_n$.
```

We can also go the other way:

```{proposition}
If $D$ is a martingale difference sequence with respect to $X$, then $Y_n = \sum_{i=1}^{n}{D_i}$ defines a martingale with respect to $X$.
```

```{proof}
$\Expect{Y_{n+1}|X_{1:n}} = \Expect{Y_n + D_{n+1}|X_{1:n}} = Y_n$.
```

```{proposition}
If $D$ is a martingale difference sequence, then $\Expect{D_n} = 0$ for all $t$.
```

```{proposition}
By mathematical induction.  Suppose $\Expect{D_n} = 0$.  By the law
of total expectation, $\Expect{D_{n+1}} = \Expect{\Expect{D_{n+1}|X_1, \ldots X_n}} =\Expect{D_n} = 0$.
```

```{proposition}
If $D$ is a martingale difference sequence, and $s < t$, then $\Expect{D_n|X_1, \ldots X_s} = 0$.
```

```{proof}
By mathematical induction and the law of total expectation, as before.
```

```{proposition}
If $D$ is a martingale difference sequence, and $s\neq t$, then $\Cov{D_n, D_m} = 0$.  That is, martingale difference sequences are uncorrelated.
```

```{proof}
Without loss of generality, say that $m < n$.  Because $\Expect{D_n} = \Expect{D_m} = 0$ (by the previous proposition), we're done if we can show $\Expect{D_m D_n} = 0$.  To do this, use the law of total expectation again:
\begin{eqnarray}
\Expect{D_m D_n} & = & \Expect{D_m \Expect{D_n|X_1, \ldots X_s}}\\
& = & \Expect{D_m 0} = 0
\end{eqnarray}
```

Because uncorrelated variables are sometimes called "orthogonal", we sometimes
say martingale difference sequences are "orthogonal sequences"[^ortho].

[^ortho]: Suppose the sequence $O$ has $\Expect{O_n} = 0$ for all $n$, and $\Cov{O_n, O_m} = 0$ for all $m\neq n$.  Is this **orthogonal sequence** $O$ necessarily a martingale difference sequence?  If so, can you prove it?  If not, can you find a counter-example, a sequence that is orthogonal but is not a martingale difference sequence?


## Sub-Gaussian martingale difference sequences

```{proposition}
Suppose $D$ is a martingale difference sequence with respect to $X$, and each $D_n$ is conditionally sub-Gaussian, meaning that $\Expect{e^{t D_{n}}|X_1, \ldots X_{n-1}} \leq \myexp{t^2 \sigma^2_n/2}$ for some $\sigma^2_n < \infty$. Then the sum of the $D_n$s is sub-Gaussian, and the sub-Gaussian constants add:
\begin{equation}
\Expect{\myexp{t \sum_{i=1}^{n}{D_i}}} \leq \myexp{-\frac{t^2}{2}\sum_{i=1}^{n}{\sigma^2_i}}
\end{equation}
```

```{proof}
By mathematical induction.  To get the inductive step, assume this holds for $n-1$, and then use the law of total expectation and the conditional sub-Gaussian bound:
\begin{eqnarray}
\Expect{\myexp{t \sum_{i=1}^{n}{D_i}}} & = & \Expect{\Expect{\myexp{t \sum_{i=1}^{n}{D_i}}|X_{1:n-1}}}\\
& = & \Expect{\left(\myexp{t \sum_{i=1}^{n-1}{D_i}}\right)\Expect{\myexp{t D_n}|X_{1:n-1}}}\\
& \leq & \Expect{\left(\myexp{t\sum_{i=1}^{n-1}{D_i}}\right) \myexp{\frac{t^2}{2\sigma^2_n}}}\\
& = & \myexp{\frac{t^2}{2}\sigma^2_n} \Expect{\myexp{t\sum_{i=1}^{n-1}{D_i}}}
\end{eqnarray}
The $n=1$ case is guaranteed by the assumption, since when $n=1$ we can't condition on earlier $X$s, hence $\Expect{e^{t D_1}} \leq \myexp{t^2 \sigma^2_1/2}$ for some $\sigma^2_1$.
```

```{proposition}
If $D$ is a martingale difference sequence with respect to $X$, and $\Prob{a_n \leq D_n \leq b_n} = 1$, then
\begin{equation}
\Expect{\myexp{t \sum_{i=1}^{n}{D_i}}} \leq \myexp{\frac{t^2}{2}\sum_{i=1}^{n}{\frac{(b_i - a_i)^2}{4}}}
\end{equation}
```

```{proof}
Combine the previous proposition with Hoeffding's bound on the moment generating function of a limited-range random variable.
```

Both of the last two propositions are sometimes called "the Azuma-Hoeffding
bound", as are a number of other closely related results about the moment
generating functions of sums of martingale differences, of martingales, on the
deviations of sums of martingale differences from their expectations, etc.  (Hoeffding's original paper indicated that his result should apply not just to independent variables but also to martingale differences, and Azuma seems to have been the first to work through the details.)

## Turning ordinary functions into martingales, and martingale differences, a.k.a. Doob's martingale

Take any function $f$ of $n$ arguments, say $f(X_1, \ldots X_n)$.  A classic,
widely-used construction relates this to a martingale.  Define
\begin{equation}
Z_k = \Expect{f(X_{1:n})|X_{1:k}}
\end{equation}
with the understanding that $Z_n = f(X_{1:n})$, the random function itself,
and that $Z_0 = \Expect{f(X_{1:n})}$, the unconditional expectation.  Let's
check that this is a martingale with respect to $X$:
- For each $k$, $Z_k$ is clearly a function of $X_{1:k}$.
- We thus need to check the conditional expectation property, that $\Expect{Z_{k+1}|X_{1:k}} = Z_k$.  But this follows by the law of total expectation:
\begin{eqnarray}
\Expect{Z_{k+1}|X_{1:k}} & = & \Expect{\Expect{f(X_1, \ldots X_n)|X_{1:k+1}}|X_{1:k}}\\
& = & \Expect{f(X_1, \ldots X_n)|X_{1:k}}\\
& = & Z_k
\end{eqnarray}
by the definition of $Z_k$.

If we define $D_k = Z_{k} - Z_{k-1}$, then $D$ is a martingale difference
sequence.  We then have the **Doob representation**
\begin{equation}
f(X_1, \ldots X_n) = \Expect{f(X_1, \ldots X_n)} + \sum_{i=1}^{n}{D_i}
\end{equation}
which shows how the function is equal to its expected value plus a sum of
expectation-0, uncorrelated random terms.

# The Efron-Stein inequality

Let us begin with the Doob representation, which to recall is
\begin{equation}
f(X_1, \ldots X_n) = \Expect{f(X_1, \ldots X_n)} + \sum_{i=1}^{n}{D_i}
\end{equation}
where the martingale difference sequence $D$ is defined by $D_k = \Expect{f(X_{1:n})|X_{1:k}} - \Expect{f(X_{1:n})|X_{1:k-1}}$.  Because $\Expect{f(X_1, \ldots X_n)}$ is non-random,
\begin{equation}
\Var{f(X_1, \ldots X_n)} = \Var{\sum_{i=1}^{n}{D_i}}
\end{equation}
Since, as we've seen, the terms in a martingale difference sequence are uncorrelated,
\begin{equation}
\Var{f(X_1, \ldots X_n)} = \sum_{i=1}^{n}{\Var{D_i}} = \sum_{i=1}^{n}{\Expect{D_i^2}}
\end{equation}
where the last equality holds because $\Expect{D_k}=0$ for all $k$.

Nothing in the last paragraph relies on the $X$ variables being independent, but
the next steps in the argument rely crucially on this.


# The bounded difference / McDiarmid inequality

Recall from the slides that we say a function $f(x_1, \ldots x_n)$ has the **bounded difference property**, with bounds $d_i$, if changing argument $x_i$ can change the value of the function by at most $d_i$:
\begin{equation}
\max_{x_i, y_i}|f(x_1, \ldots x_{i-1}, x_i, x_{i+1}, \ldots x_n) - f(x_1, \ldots x_{i-1}, y_i, x_{i+1}, \ldots x_n)| \leq d_i
\end{equation}

We want to use this to show that $f$ must be close to its expectation.  Doob's
construction tells us how to re-write $f$ as a sum of martingale differences,
and the Azuma-Hoeffding result tells us sums of martingale differences are
close to their expectations.  So we're almost there, and just need to prove
the following:

```{proposition}
If $f$ has the bounded difference property with bounds $d$, then the martingale differences $D_k \equiv \Expect{f(X_{1:n})|X_{1:k}} - \Expect{f(X_{1:n})|X_{1:k-1}}$ is bounded to an interval of the same length $d_k$.
```


```{proof}
Let's define upper and lower limits for $D_k$:
\begin{eqnarray}
L_k & \equiv & \min_{x}{\Expect{f(X_{1:n})|X_1, \ldots X_{k-1}, X_k=x} - \Expect{f(X_{1:n})|X_1, \ldots X_{k-1}}}\\
U_k & \equiv & \max_{x}\Expect{f(X_{1:n})|X_1, \ldots X_{k-1}, X_k=x} - \Expect{f(X_{1:n})|X_1, \ldots X_{k-1}}
\end{eqnarray}

Notice that $L_k$ and $U_k$ are random variables, since they are functions
of $X_{1:k-1}$ (hence they're written with upper-case letters).  Nonetheless
it should be clear that
\begin{equation}
L_k \leq D_k \leq U_k
\end{equation}
so we need to show that $U_k - L_k \leq d_k$.
\begin{eqnarray}
U_k - L_k & = & \max_{x}{\Expect{f(X_{1:n})|X_{1:k-1}, X_k=x}} - \min_{x^{\prime}}{\Expect{f(X_{1:n})|X_{1:k-1}, X_k=x^{\prime}}}\\
& = & \max_{x, x^{\prime}}{\Expect{f(X_{1:n})|X_{1:k-1}, X_k=x} - \Expect{f(X_{1:n})|X_{1:k-1}, X_k=x^{\prime}}}\\
& = & \max_{x, x^{\prime}}{\int{f(X_1, \ldots X_{k-1}, x, x_{k+1}, \ldots x_n) p(x_{k+1:n}|X_{1:k-1}, X_k=x)dx_{k+1:n}}}\\
\nonumber & & - \int{f(X_1, \ldots X_{k-1}, x, x_{k+1}, \ldots x_n) p(x_{k+1:n|X_{1:k-1}, X_k=x^{\prime}})dx_{k+1:n}}
\end{eqnarray}
(Of course if the $X$s are discrete we do a sum instead of this integral.)
Here, finally, we use the fact that the $X_k$ are independent, so that the distribution of $X_{k+1}, \ldots X_n$ is the same no matter what $X_{1:k}$ might be, and in particular whether $X_k=x$ or $X_k=x^{\prime}$ makes no difference.  This lets us combine the integrals:
\begin{eqnarray}
\lefteqn{U_k - L_k } & &\\
\nonumber & = & \max_{x, x^{\prime}}{\int{\left(f(X_1, \ldots X_{k-1}, x, x_{k+1}, \ldots x_n) - f(X_1, \ldots X_{k-1}, x, x_{k+1}, \ldots x_n)\right) p(x_{k+1:n})  dx_{k+1:n}}}\\
& \leq & \int{d_k p(x_{k+1:n})  dx_{k+1:n}}\\
& = & d_k
\end{eqnarray}
using the bounded difference property of $f$.  So we have shown that $\Prob{U_k
\geq D_k \geq L_k} = 1$ and $U_k - L_k \leq d_k$, as desired.
```

Combining this with our previous result about bounded martingale differences,
we get

```{proposition}
If $f$ has the bounded difference property with bounds $d$, then
\begin{equation}
\Expect{\myexp{t f(X_{1:n})}} \leq \myexp{\frac{t^2}{2}\sum_{i=1}^{n}{\frac{d_i^2}{4}}}
\end{equation}
```

and the bounded-difference or McDiarmid inequality in the slides follows
by the now-familiar sub-Gaussian deviation bound.

# References

