\[ \newcommand{\Yhat}{\hat{Y}} \newcommand{\Prob}[1]{\mathbb{P}\left( #1 \right)} \newcommand{\indep}{\perp} \]

My presentation of this topic largely (but not entirely) follows the excellent review paper by Corbett-Davies and Goel (2018).

When we talk about classification problem, we always mean that we’re trying to predict a categorical, usually binary, label, outcome or class \(Y\) from features \(X\) (which may or may not be categorical).

We’ll be a little more telegraphic today.

“Protected attributes”

Some notions of “fairness” for classification

  1. Don’t use protected features directly
    1. Sometimes called “anti-classification”
    2. What about strongly-associated unprotected features?
  2. Have equal measures of error across groups
    1. Sometimes called “classification parity”
    2. Which error measures, exactly?
  3. Calibration: everyone with the same score should have the same actual probability of \(Y=1\), regardless of group
    1. Conditional independence of \(Y\) from protected attribute given score
    2. Weak!

Concrete use-case: Pretrial detention, recidivism prediction

  • You don’t get arrested, your screw-up cousin Archie gets arrested
  • Court decides whether or not to keep Archie in jail pending trial or let Archie go (perhaps on bail)
  • Court wants Archie to show up and not do any more crimes
    • \(Y=1\): Archie will be arrested for another crime if released
    • \(Y=0\): Archie will not be arrested
    • Similarly for failure to appear on trial date, arrest for violence, etc.


  • Archie’s features \(=X = (X_p, X_u)\) where \(X_p\) are the protected features and \(X_u\) are the unprotected ones
  • \(Y=\) whether or not Archie will be arrested for another crime before trial
    • Or: will show up for trial, will be re-arrested after being released from prison, will default on the loan, …
    • Generally, today, \(Y=1\) is the bad case
  • \(\Yhat(x) =\) prediction we make about someone with features \(x\)
    • Here \(\Yhat=1\) means “we predict re-arrest” (or recidivism), and so someone paying attention to use would presumably not release this person
    • \(\Yhat(x)\) can ignore some features in \(x\)!
  • \(p(x) = \Prob{Y=1|X=x}\) is the true risk function
    • Cost-minimizing \(\Yhat(x)\) thresholds based on \(p(x)\), and the costs of both errors
    • Note that true risk function isn’t known
  • \(s(x) =\) risk score we calculate based on \(x\) (which may or may not be an estimate of \(p(x)\))


“Classification parity”



The lurking problem: designedly missing data

Summing up

Backup: Filling in Chouldechova’s “it is straightforward to show that”

(maybe it’s straightforward for Alex…)

I’ll write out the algebra for the population as a whole; doing it for each group just means sprinkling in conditioning signs.

\(R = \Prob{Y=1}\) is the true prevalence or base rate.

Chouldechova’s claim is that \[ FPR = \frac{R}{1-R} \frac{1-PPV}{PPV} (1-FNR) \]

Substituting in from the definitions, \[ \Prob{\Yhat=1|Y=0} = \frac{\Prob{Y=1}}{\Prob{Y=0}} \frac{1-\Prob{Y=1|\Yhat=1}}{\Prob{Y=1|\Yhat=1}} (1-\Prob{\Yhat=0|Y=1}) \] Since \(Y\) and \(\Yhat\) are both binary, \[ \Prob{\Yhat=1|Y=0} = \frac{\Prob{Y=1}}{\Prob{Y=0}} \frac{\Prob{Y=0|\Yhat=1}}{\Prob{Y=1|\Yhat=1}}\Prob{\Yhat=1|Y=1} \] but \[\begin{eqnarray} \Prob{Y=0|\Yhat=1} & = & \Prob{Y=0, \Yhat=1}/\Prob{\Yhat=1}\\ &= & \Prob{\Yhat=1|Y=0}\Prob{Y=0}/\Prob{\Yhat=1}\\ \Prob{Y=1|\Yhat=1} & = & \Prob{\Yhat=1|Y=1}\Prob{Y=1}/\Prob{\Yhat=1} \end{eqnarray}\] so \[\begin{eqnarray} \Prob{Y=0|\Yhat=1} / \Prob{Y=1|\Yhat=1} & = & \Prob{\Yhat=1|Y=0}\Prob{Y=0} / \Prob{\Yhat=1|Y=1} \Prob{Y=1}\\ \frac{\Prob{Y=1}}{\Prob{Y=0}} \frac{\Prob{Y=0|\Yhat=1}}{ \Prob{Y=1|\Yhat=1}} & = &\Prob{\Yhat=1|Y=0} / \Prob{\Yhat=1|Y=1} \end{eqnarray}\] and so, substituting in, we get \[ \Prob{\Yhat=1|Y=0} = \Prob{\Yhat=1|Y=0} \] which is certainly true.


Corbett-Davies, Sam, and Sharad Goel. 2018. “The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning.” E-print, arxiv:1808.00023. https://arxiv.org/abs/1808.00023.

DeDeo, Simon. 2016. “Wrong Side of the Tracks: Big Data and Protected Categories.” In Big Data Is Not a Monolith, edited by Cassidy R. Sugimoto, Hamid R. Ekbia, and Michael Mattioli, 31–42. Cambridge, Massachusetts: MIT Press. http://arxiv.org/abs/1412.4643.