Comparing methods of detecting and segmenting unruptured intracranial aneurysms on TOF-MRAS: The ADAM challenge
Kimberley M. Timmins,
Irene C. van der Schaaf,
Edwin Bennink,
Ynte M. Ruigrok,
Xingle An,
Michael Baumgartner,
Pascal Bourdon,
Riccardo De Feo,
Tommaso Di Noto,
Florian Dubost,
Augusto Fava-Sanches,
Xue Feng,
Corentin Giroud,
Inteneural Group,
Minghui Hu,
Paul F. Jaeger,
Juhana Kaiponen,
Michał Klimont,
Yuexiang Li,
Hongwei Li,
Yi Lin,
Timo Loehr,
Jun Ma,
Klaus H. Maier-Hein,
Guillaume Marie,
Bjoern Menze,
Jonas Richiardi,
Saifeddine Rjiba,
Dhaval Shah,
Suprosanna Shit,
Jussi Tohka,
Thierry Urruty,
Urszula Walińska,
Xiaoping Yang,
Yunqiao Yang,
Yin Yin,
Birgitta K. Velthuis,
Hugo J. Kuijf
Affiliations
Kimberley M. Timmins
Image Sciences Institute, University Medical Center Utrecht, Utrecht, the Netherlands; Corresponding author.
Irene C. van der Schaaf
Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands
Edwin Bennink
Image Sciences Institute, University Medical Center Utrecht, Utrecht, the Netherlands
Ynte M. Ruigrok
Department of Neurology and Neurosurgery, UMC Utrecht Brain Center, University Medical Center Utrecht, Utrecht, the Netherlands
Xingle An
China Electronics Cloud Brain (Tianjin) Technology CO., LTD, 300309 PR China
Michael Baumgartner
Division of Medical Image Computing, German Cancer Research Center, Heidelberg, Germany
Pascal Bourdon
Xlim Laboratory, University of Poitiers, UMR CNRS 7252, Poitiers; Poitiers University Hospital, CHU, Poitiers
Riccardo De Feo
Sapienza Università di Roma, 00184 Rome Italy; A.I. Virtanen Institute for Molecular Sciences, University of Eastern Finland, 70210 Kuopio, Finland
Tommaso Di Noto
Department of Radiology, Lausanne University Hospital; University of Lausanne, Rue du Bugnon 46, Lausanne, CH, 1011
Florian Dubost
Zelos Mediacorp, Rotterdam, the Netherlands
Augusto Fava-Sanches
Institute of Neuroradiology, University Hospital LMU, Munich, Germany
Xue Feng
University of Virginia, Biomedical Engineering, Thornton Hall, P.O. Box 400259, Charlottesville, VA, USA, 22904-4259
Corentin Giroud
Zelos Mediacorp, Rotterdam, the Netherlands
Inteneural Group
Inteneural Networks, Warsaw, Poland
Minghui Hu
Union Strong (Beijing) Technology Co. Ltd., DaZu Plaza T3-901, No. 2 Ronghua South Road, Beijing Economic Technological Development Area, Beijing, China, 100176
Paul F. Jaeger
Division of Medical Image Computing, German Cancer Research Center, Heidelberg, Germany
Juhana Kaiponen
A.I. Virtanen Institute for Molecular Sciences, University of Eastern Finland, 70210 Kuopio, Finland
Michał Klimont
Inteneural Networks, Warsaw, Poland; Department of Radiology, Poznań University of Medical Sciences, Poznań, Poland
Yuexiang Li
Tencent Jarvis Lab, Shenzhen, China
Hongwei Li
Department of Computer Science, Technical University of Munich; Department of Quantitative Biomedicine, University of Zurich.
Yi Lin
Tencent Jarvis Lab, Shenzhen, China
Timo Loehr
Department of Computer Science, Technical University of Munich
Jun Ma
Department of Mathematics, Nanjing University of Science and Technology, Nanjing, 210094 PR China
Klaus H. Maier-Hein
Division of Medical Image Computing, German Cancer Research Center, Heidelberg, Germany; Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany
Guillaume Marie
Department of Radiology, Lausanne University Hospital; University of Lausanne, Rue du Bugnon 46, Lausanne, CH, 1011
Bjoern Menze
Department of Quantitative Biomedicine, University of Zurich.; Department of Computer Science, Technical University of Munich
Jonas Richiardi
Department of Radiology, Lausanne University Hospital; University of Lausanne, Rue du Bugnon 46, Lausanne, CH, 1011
Saifeddine Rjiba
Canon Medical Systems, France; Xlim Laboratory, University of Poitiers, UMR CNRS 7252, Poitiers
Dhaval Shah
Department of Informatics, Technische Universität München, Munich, Germany
Suprosanna Shit
Department of Informatics, Technische Universität München, Munich, Germany
Jussi Tohka
A.I. Virtanen Institute for Molecular Sciences, University of Eastern Finland, 70210 Kuopio, Finland
Thierry Urruty
Xlim Laboratory, University of Poitiers, UMR CNRS 7252, Poitiers; Poitiers University Hospital, CHU, Poitiers
Urszula Walińska
Inteneural Networks, Warsaw, Poland
Xiaoping Yang
Department of Mathematics, Nanjing University of Science and Technology, Nanjing, 210094 PR China
Yunqiao Yang
Huazhong University of Science and Technology, Wuhan, China
Yin Yin
Union Strong (Beijing) Technology Co. Ltd., DaZu Plaza T3-901, No. 2 Ronghua South Road, Beijing Economic Technological Development Area, Beijing, China, 100176
Birgitta K. Velthuis
Department of Radiology, University Medical Center Utrecht, Utrecht, the Netherlands
Hugo J. Kuijf
Image Sciences Institute, University Medical Center Utrecht, Utrecht, the Netherlands
Accurate detection and quantification of unruptured intracranial aneurysms (UIAs) is important for rupture risk assessment and to allow an informed treatment decision to be made. Currently, 2D manual measures used to assess UIAs on Time-of-Flight magnetic resonance angiographies (TOF-MRAs) lack 3D information and there is substantial inter-observer variability for both aneurysm detection and assessment of aneurysm size and growth. 3D measures could be helpful to improve aneurysm detection and quantification but are time-consuming and would therefore benefit from a reliable automatic UIA detection and segmentation method. The Aneurysm Detection and segMentation (ADAM) challenge was organised in which methods for automatic UIA detection and segmentation were developed and submitted to be evaluated on a diverse clinical TOF-MRA dataset.A training set (113 cases with a total of 129 UIAs) was released, each case including a TOF-MRA, a structural MR image (T1, T2 or FLAIR), annotation of any present UIA(s) and the centre voxel of the UIA(s). A test set of 141 cases (with 153 UIAs) was used for evaluation. Two tasks were proposed: (1) detection and (2) segmentation of UIAs on TOF-MRAs. Teams developed and submitted containerised methods to be evaluated on the test set. Task 1 was evaluated using metrics of sensitivity and false positive count. Task 2 was evaluated using dice similarity coefficient, modified hausdorff distance (95th percentile) and volumetric similarity. For each task, a ranking was made based on the average of the metrics.In total, eleven teams participated in task 1 and nine of those teams participated in task 2. Task 1 was won by a method specifically designed for the detection task (i.e. not participating in task 2). Based on segmentation metrics, the top two methods for task 2 performed statistically significantly better than all other methods. The detection performance of the top-ranking methods was comparable to visual inspection for larger aneurysms. Segmentation performance of the top ranking method, after selection of true UIAs, was similar to interobserver performance. The ADAM challenge remains open for future submissions and improved submissions, with a live leaderboard to provide benchmarking for method developments at https://adam.isi.uu.nl/.
