ECAS-ENBIS Course: Conformal Prediction: How to quantify uncertainty of machine learning models?

Sunday 10 Sept 2023, 14:00 → 18:00 Europe/Amsterdam

ECAS-ENBIS Course: Conformal Prediction: How to quantify uncertainty of machine learning models?

Part of the ENBIS-23 Valencia conference.

This half-day course is a joint initiative from ENBIS and ECAS (http://ecas.fenstats.eu/) which has provided courses since 1987 to achieve training in special areas of statistics for both researchers and teachers for universities and professionals in industry fields.

Instructor

Margaux Zaffran (INRIA, Ecole Polytechnique and EDF, France)

Overview

By leveraging increasingly large data sets, statistical algorithms and machine learning methods can be used to support high-stakes decision-making problems such as autonomous driving, energy, medical or civic applications, and more. In order to ensure the safe deployment of predictive models, it is crucial to quantify the uncertainty of the resulting predictions, communicating the limits of predictive performance. Therefore, uncertainty quantification attracts a lot of attention in recent years, particularly methods that are based on Conformal Prediction. Conformal Prediction provides controlled predictive regions for any underlying predictive algorithm (e.g., neural networks and random forests), in finite samples with no assumption on the data distribution except for the exchangeability of the train and test data. Conformal Prediction has already been successfully used to predict in real time the results of the last US presidential elections (2020) by the Washington Post.

This short course is a first introduction to Conformal Prediction, aimed at a broad audience of practitioners and researchers. It requires basic knowledge in statistics, probability and machine learning (including regression, classification, training and validation splitting strategies, for example, but no prior knowledge in any specific machine learning algorithms is required). Available software and implementations of Conformal Prediction will be reviewed.

We will introduce the Conformal Prediction framework, in its generic version, applicable to both regression and classification tasks. Then, we will review the existing challenges, such as the computational cost, conditional and adaptive coverage or even distribution shifts. and the recent solutions that have been proposed to handle them. Finally, we will focus on one specific challenge: time series forecasting. Temporal data are not exchangeable, therefore they do not met the only assumption required by Conformal Prediction. We will highlight new developments on this topic, and illustrate these procedures on the task of forecasting French electricity prices.

Outline

Below is given an outline of the course, with the main references for each part.

Part 1: Introduction to conformal prediction

• Reminders on machine learning, in particular supervised learning, and dive into quantile regression
• Split conformal prediction: from standard regression to classification
• A generalized framework: full conformal prediction

Part 2: Challenges

• Tradeoff between computational cost and statistical efficiency: jackknife+ and CV+ approaches
• Improving the adaptivity of conformal prediction
• Beyond the exchangeability assumption

Part 3: Adaptive Conformal Predictions for Time Series

Short bio

Margaux is a last year PhD student at both INRIA and Ecole Polytechnique (Paris), with Aymeric Dieuleveut (Ecole Polytechnique) and Julie Josse (INRIA). She also works with Electricité de France, the leading producer and supplier of electricity in France and Europe, in collaboration with Olivier Féron and Yannig Goude.

Her research interests lie in developing tools able to quantify uncertainty with any machine learning algorithm, and more generally, in probabilistic predictions, motivated by real-world applications in the energy sector. This specifity attracts her towards time series and missing data. Before focusing on conformal prediction and distribution-free uncertainty quantification, she analyzed quantile regression approaches, based on generalized additive models, at the University of Bristol.

https://mzaffran.github.io

1 ECAS-ENBIS Course: Conformal Prediction: How to quantify uncertainty of machine learning models?

Speaker: Margaux Zaffran