Multilevel Multimodal Framework for Automatic Collateral Scoring in Brain Stroke

Abstract

In patients with ischemic brain stroke, collateral circulation plays a crucial role in selecting patients suitable for endovascular therapy. The presence of well-developed collaterals improves the patient’s chances of recovery. In clinical practice, the presence of collaterals is diagnosed on a Computed Tomography Angiography scan. The radiologist grades it on the basis of subjective visual assessment, which is prone to interobserver and intraobserver variability. Computer-based methods of collateral assessment face the challenge of non-uniform scan volume, leading to manual selection of slices, meaning that the most imperative slices have to be manually selected by the radiologist. This paper proposes a multilevel multimodal hierarchical framework for automated collateral scoring. Specifically, we propose deploying a Convolutional Neural Network for image selection based on the visibility of collaterals and a multimodal model for comparing the occluded and contralateral sides of the brain for collateral scoring. We also generate a patient-level prediction by integrating automated machine learning in the proposed framework. While the proposed multimodal predictor contributes to Artificial Intelligence, the proposed end-to-end framework is an application in engineering. The proposed framework has been trained and tested on 116 patients, with five-fold cross-validation, achieving an accuracy of 91.17% for multi-class collateral scores and 94.118% for binary class collateral scores. The proposed multimodal predictor achieved a weighted F1 score of 0.86 and 0.95 on multi-class and binary-class collateral scores, respectively. The proposed framework is fast, efficient, and scalable for real-world deployments. Automated evaluation of collaterals with attention maps for explainability would complement radiologists’ efforts. Code for the proposed framework is available at: https://github.com/rishiraj-cs/collaterals_ML_MM.

Keywords

• Deep learning
• biomedical image processing
• medical expert systems
• stroke (medical condition)