The Machines Aren’t Taking Over (Yet): An Empirical Comparison of Traditional, Profiling, and Machine Learning Approaches to Criterion-Related Validation

Abstract

Criterion-related validation (CRV) studies are used to demonstrate the effectiveness of selection procedures. However, traditional CRV studies require significant investment of time and resources, as well as large sample sizes, which often create practical challenges. New techniques, which use machine learning to develop classification models from limited amounts of data, have emerged as a more efficient alternative. This study empirically investigates the effectiveness of traditional CRV with a variety of profiling approaches and machine learning techniques using repeated cross-validation. Results show that the traditional approach generally performs best both in terms of predicting performance and larger group differences between candidates identified as top or non-top performers. In addition to empirical effectiveness, other practical implications are discussed.