External Validation of a Machine Learning Model for Schizophrenia Classification

Abstract

Background and Objective: Excellent generalizability is the precondition for the widespread practical implementation of machine learning models. In our previous study, we developed the schizophrenia classification model (SZ classifier) to identify potential schizophrenia patients in the Japanese population. The SZ classifier has exhibited impressive performance during internal validation. However, ensuring the robustness and generalizability of the SZ classifier requires external validation across independent sample sets. In this study, we aimed to present an external validation of the SZ classifier using outpatient data. Methods: The SZ classifier was trained by using online survey data, which incorporate demographic, health-related, and social comorbidity features. External validation was conducted using an outpatient sample set which is independent from the sample set during the model development phase. The model performance was assessed based on the sensitivity and misclassification rates for schizophrenia, bipolar disorder, and major depression patients. Results: The SZ classifier demonstrated a sensitivity of 0.75 when applied to schizophrenia patients. The misclassification rates were 59% and 55% for bipolar disorder and major depression patients, respectively. Conclusions: The SZ classifier currently encounters challenges in accurately determining the presence or absence of schizophrenia at the individual level. Prior to widespread practical implementation, enhancements are necessary to bolster the accuracy and diminish the misclassification rates. Despite the current limitations of the model, such as poor specificity for certain psychiatric disorders, there is potential for improvement if including multiple types of psychiatric disorders during model development.

Keywords

• external validation
• schizophrenia
• bipolar
• depression
• machine learning
• classification