Performance of Artificial Intelligence Models (Bidirectional Encoder Representations from Transformers + TextCNN) in Detecting Eight Psychiatric Diagnoses from Unstructured Texts Chinese Electronic Medical Records

Abstract

Objectives: Advances in artificial intelligence (AI) have revolutionized various industries, including health care. In this study, we intended to explore the capability of AI assistants in psychiatric diagnoses. To achieve this goal, we proposed a series of deep active learning models, namely bidirectional encoder representations from transformers (BERT) – TextCNN. These models combine the strengths of two powerful techniques: BERT and convolutional neural network (CNN) for the text. Methods: We collected 21,003 Chinese psychiatry electronic medical records (EMRs) and developed two types of models: a multi-diagnosis classifier and eight single-diagnosis classifiers for schizophrenia (SCZ), major depressive disorder (MDD), manic state (MANIA), adjustment disorder (ADJ), substance use disorder (SUD), personality disorder (PD), attention-deficit/hyperactivity disorder (ADHD), and autistic spectrum disorder (ASD). Their performance was compared through plotting receiver operating characteristic curves and assessing the performance, area under curve (AUC) using the DeLong test. Results: This study showed the excellent performance of our BERT + TextCNN models in detecting almost all eight psychiatric diagnoses, achieving AUCs being greater than 0.9, except for the single-diagnosis classifier for ADHD (AUC = 0.83). Conclusion: This study highlights the promising applicability of the BERT + TextCNN model as a diagnostic assistant for psychiatry diagnoses derived from EMRs. Being consistent with previous findings, the single-diagnosis classifiers generally outperform the multi-diagnosis classifier in predicting most diagnoses, though not all. Further studies are warranted to confirm whether the specific characteristics of illnesses contribute to the performance gap between multi- and single-diagnosis classifiers.

Keywords

• area under curve; natural language processing; the international classification of diseases; 9th version; traditional chinese characters