Determining Novice and Expert Status in Human–Automation Interaction Through Hidden Markov Models

Abstract

Detecting when operators achieve expert proficiency is critical for organizations that employ human–automation interaction (HAI) in operations, particularly in safety-critical settings. Training operators for complex systems demand substantial time and resources, necessitated by safety considerations and the expansive scale of these systems. Recognizing operator expertise becomes instrumental in resource optimization and training efficiency. This study explores a modeling framework for real-time analysis of HAI operator behavior and strategies. Proposing a departure from traditional assessments, the research advocates for leveraging hidden Markov models (HMMs) to provide a comprehensive portrayal of operator performance, facilitating a nuanced comparison of expert and novice strategies. Using data from a real-time strategy game, the paper details the development of HMMs and elucidates training and interface design implications. Results affirm the hypothesis that experts formulate more efficient strategies, reflected in HMMs with fewer hidden states compared to those describing novice behavior. This aligns with prior research emphasizing the organized nature of experts’ strategies. In-depth analysis delves into specific states, frequencies, and predominant strategies, revealing distinctions between experts’ offensive focus and novices’ emphasis on initial setup aspects of gameplay.