Speaker
Description
In recent decades, machine learning and industrial statistics have moved closer to each other. CQM, a consultancy company, performs projects in supply chains, logistics, and industrial R&D that often involve building prediction models using techniques from machine learning. For these models, challenges persist, e.g. if the dataset is small, has a group structure, or is a time series. At the same time, confidence intervals for the performance of the prediction model are often ignored, whereas it can be very valuable to assess model in a business context.
For assessing the performance of the prediction model, we encounter several challenges. 1) Choice between two common strategies: either training set - test set split or k-fold CV (cross validation), both with pros and cons. 2) A proper statistical description of k-fold CV has several subtleties in terms of bias, estimation, dependency. 3) Some useful metrics such as Area under Curve are just outside the scope of many texts on k-fold CV. 4) There are several strategies for computing a confidence interval. Strategies can be basic, and more advanced methods are found in recent literature with formal coverage guarantees, addressing the computational burden, or acknowledging group structure in data. We performed a study into these issues, giving clarity and resulting in practical guidelines to navigate the these challenges.
This study was carried out as part of the European project ASIMOV (https://www.asimov-project.eu/)
| Type of presentation | Talk |
|---|---|
| Classification | Mainly methodology |
| Keywords | cross validation, machine learning, statistics |