Aaditya Ramdas – Distribution-free uncertainty quantificationIt is of great practical interest to quantify the uncertainty of predictions from regression and classification algorithms used in machine learning. One such framework that allows us to produce calibrated prediction sets with no distributional assumptions (beyond exchangeability of the data) is called conformal prediction, and has been studied by Vladimir Vovk and colleagues since 2000 or so. At a very high level, the idea involves training the regression algorithm multiple times and using either the in-sample or out-of-sample training residuals within the training or holdout set to quantify and calibrate the uncertainty of the prediction at a new point. Remarkably, the validity of the method does not depend on which ML algorithm was used, black box or not, or any other model assumptions. Naturally the length or size of the prediction set will depend on these factors, but not correctness. Another interesting problem is that of classifier calibration, which can also be achieved without distributional assumptions. I am actively working on theoretical and practical problems within this area. You can also find some software packages linked below. Distribution-free uncertainty quantification (conformal, calibration) (package 1) (package 2) (tutorial)
|