Precision Structuring of Free-Text Surgical Record for Enhanced Stroke Management: A Comparative Evaluation of Large Language Models

Abstract

Mengfei Wang,1 Jianyong Wei,2 Yao Zeng,3 Lisong Dai,4 Bicong Yan,4 Yueqi Zhu,4 Xiaoer Wei,4 Yidong Jin,1 Yuehua Li4 1School of Health Science and Engineering, University of Shanghai for Science and Technology, Shanghai, People’s Republic of China; 2Clinical Research Center, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, People’s Republic of China; 3Shenyang University of Technology, Shenyang, Liaoning, People’s Republic of China; 4Institute of Diagnostic and Interventional Radiology, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, People’s Republic of ChinaCorrespondence: Yuehua Li, Institute of Diagnostic and Interventional Radiology, Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, No. 600 Yishan Road, Shanghai, 200233, People’s Republic of China, Email [email protected]: Mechanical thrombectomy (MTB) is a critical procedure for acute ischemic stroke (AIS) patients. However, the free-text format of MTB surgical records limits the formulation of effective postoperative patient management and rehabilitation plans. This study compares the efficacy of large language models (LLMs) in structuring data from these free-text MTB surgical record.Methods: This retrospective study collected a total of 382 MTB surgical records from a tertiary hospital. An initial analysis of 30 surgical record from these records provided a guiding prompt for LLMs, focusing on basic and advanced characteristics, such as occlusion locations, thrombectomy maneuvers, reperfusion status, and intraoperative complications. Six LLMs—ChatGPT, GPT-4, GeminiPro, ChatGLM4, Spark3, and QwenMax—were assessed against data extracted by neuroradiologists and a junior physician for comparison. The all 382 surgical records were used to test the performance of LLMs. The performance of the LLMs was quantified using Accuracy, Sensitivity, Specificity, AUC, and MSE as an additional metric for advanced characteristics.Results: All LLMs showed high performance in characteristic extraction, achieving an average accuracy of 95.09 ± 4.98% across 48 items, and 78.05 ± 4.2% overall. GLM4 and GPT-4 were most accurate in advanced characteristics extraction, with accuracies of 84.03% and 82.20%, respectively. The processing time for LLMs averaged 73.10 ± 10.86 seconds of six models, significantly faster than the 427.88 seconds for manual extraction by physicians.Conclusion: LLMs, particularly GLM4 and GPT-4, efficiently and accurately structured both general and advanced characteristics from MTB surgical record, outperforming manual extraction methods and demonstrating potential for enhancing clinical data management in AIS treatment.Keywords: large language models, free-text report, mechanical thrombectomy, acute ischemia stroke

Keywords

• large language models
• free-text report
• mechanical thrombectomy
• acute ischemia stroke