The misdiagnosis of headache disorders is a serious issue, and AI-based headache model diagnoses with external validation are scarce. We previously developed an artificial intelligence (AI)-based headache diagnosis model using a database of 4000 patients’ questionnaires in a headache-specializing clinic and herein performed external validation prospectively. The validation cohort of 59 headache patients was prospectively collected from August 2023 to February 2024 at our or collaborating multicenter institutions. The ground truth was specialists’ diagnoses based on the initial questionnaire and at least a one-month headache diary after the initial consultation. The diagnostic performance of the AI model was evaluated. The mean age was 42.55 ± 12.74 years, and 51/59 (86.67%) of the patients were female. No missing values were reported. Of the 59 patients, 56 (89.83%) had migraines or medication-overuse headaches, and 3 (5.08%) had tension-type headaches. No one had trigeminal autonomic cephalalgias or other headaches. The models’ overall accuracy and kappa for the ground truth were 94.92% and 0.65 (95%CI 0.21–1.00), respectively. The sensitivity, specificity, precision, and F values for migraines were 98.21%, 66.67%, 98.21%, and 98.21%, respectively. There was disagreement between the AI diagnosis and the ground truth by headache specialists in two patients. This is the first external validation of the AI headache diagnosis model. Further data collection and external validation are required to strengthen and improve its performance in real-world settings.
