Balanced Leave-One-Subject-Out Cross- Validation for Microsleep Classification

Abstract

Inter-individual differences in the feature distribution of electroencephalograms (EEG) during microsleep (MS) raise questions about the generalizability of the methodology, because methodology should have the same validity for subjects to be included in the analysis in the future as for subjects included so far. We address this question using leave-one-subject- out cross-validation (LOSO CV) to simulate inclusion of test data from future subjects. Investigations are based on EEG of 70 subjects across four studies conducted in our driving simulation lab. 9,297 MS and 10,264 counter-examples of sustained attention (SA) were included in the analysis. Three variants of sample balancing are compared: (1) none, (2) overall balancing, and (3) between-subjects balancing. Resulting classification accuracies range from 80% to 100% for 67 subjects, but for 3 subjects they are at 66.5%, 73.1%, 78.3%. Averaged across subjects, accuracies were (1) 90.2%, (2) 90.2% and (3) 89.1%. Thus, balancing might not be essential if LOSO CV is performed over a sufficiently large number of subjects. Results of standard CV must be regarded as optimistically biased.

Keywords

• electroencephalogram
• microsleep
• machine learning
• cross validation
• gradient boosting machine