Speakers
Description
An important issue in classification problems is the comparison of classifiers predictive performance, commonly measured as proportion of correct classifications and often referred to as accuracy or similarity measure.
This paper suggests a two-step fixed-sequence approach in order to identify the best performing classifiers among those selected as suitable for the problem at hand. At the first step of the fixed-sequence approach, the hypothesis that each classifier accuracy exceeds a desired performance threshold is tested via a simultaneous inference procedure accounting for the joint distribution of individual test statistics and the correlation between them. At the second step, focusing only on classifiers selected at first step, significant performance differences are investigated via a homogeneity test.
The applicability and usefulness of the two-step approach is illustrated through two real case studies concerning nominal and ordinal multi-class classification problems. The accuracy of three machine learning algorithms (i.e. Deep Neural Network, Random Forest, Extreme Gradient Boosting) is assessed via Gwet’s Agreement Coefficient (AC) and compared against similarity measure and Cohen Kappa. Case studies results reveal the absence of paradoxical behavior in AC coefficient and the positive effect of a weighting scheme accounting for misclassification severity with ordinal classifications, shedding light on the advantages of AC as measure of classifier accuracy.
| Keywords | Multi-class classifications, Classifier accuracy, Fixed-sequence approach |
|---|