Relationship between prediction accuracy and feature importance reliability: An empirical and theoretical study
Jianzhong Chen,
Leon Qi Rong Ooi,
Trevor Wei Kiat Tan,
Shaoshi Zhang,
Jingwei Li,
Christopher L. Asplund,
Simon B Eickhoff,
Danilo Bzdok,
Avram J Holmes,
B.T. Thomas Yeo
Affiliations
Jianzhong Chen
Centre for Sleep and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Centre for Translational MR Research, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Department of Electrical and Computer Engineering, National University of Singapore, Singapore; N.1 Institute for Health & Institute for Digital Medicine (WisDM), National University of Singapore, Singapore
Leon Qi Rong Ooi
Centre for Sleep and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Centre for Translational MR Research, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Department of Electrical and Computer Engineering, National University of Singapore, Singapore; N.1 Institute for Health & Institute for Digital Medicine (WisDM), National University of Singapore, Singapore; Integrative Sciences and Engineering Programme (ISEP), National University of Singapore, Singapore
Trevor Wei Kiat Tan
Centre for Sleep and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Centre for Translational MR Research, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Department of Electrical and Computer Engineering, National University of Singapore, Singapore; N.1 Institute for Health & Institute for Digital Medicine (WisDM), National University of Singapore, Singapore; Integrative Sciences and Engineering Programme (ISEP), National University of Singapore, Singapore
Shaoshi Zhang
Centre for Sleep and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Centre for Translational MR Research, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Department of Electrical and Computer Engineering, National University of Singapore, Singapore; N.1 Institute for Health & Institute for Digital Medicine (WisDM), National University of Singapore, Singapore; Integrative Sciences and Engineering Programme (ISEP), National University of Singapore, Singapore
Jingwei Li
Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany; Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University Düsseldorf, Düsseldorf, Germany
Christopher L. Asplund
Centre for Sleep and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Centre for Translational MR Research, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; N.1 Institute for Health & Institute for Digital Medicine (WisDM), National University of Singapore, Singapore; Division of Social Sciences, Yale-NUS College, Singapore; Department of Psychology, National University of Singapore, Singapore; Duke-NUS Medical School, Singapore
Simon B Eickhoff
Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany; Institute for Systems Neuroscience, Medical Faculty, Heinrich-Heine University Düsseldorf, Düsseldorf, Germany
Danilo Bzdok
Department of Biomedical Engineering, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada; Mila - Quebec AI Institute, Montreal, Canada
Avram J Holmes
Departments of Psychology and Psychiatry, Yale University, New Haven, CT, USA
B.T. Thomas Yeo
Centre for Sleep and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Centre for Translational MR Research, Yong Loo Lin School of Medicine, National University of Singapore, Singapore; Department of Electrical and Computer Engineering, National University of Singapore, Singapore; N.1 Institute for Health & Institute for Digital Medicine (WisDM), National University of Singapore, Singapore; Integrative Sciences and Engineering Programme (ISEP), National University of Singapore, Singapore; Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, USA; Corresponding author at: Centre for Sleep and Cognition, Yong Loo Lin School of Medicine, National University of Singapore, Singapore.
There is significant interest in using neuroimaging data to predict behavior. The predictive models are often interpreted by the computation of feature importance, which quantifies the predictive relevance of an imaging feature. Tian and Zalesky (2021) suggest that feature importance estimates exhibit low split-half reliability, as well as a trade-off between prediction accuracy and feature importance reliability across parcellation resolutions. However, it is unclear whether the trade-off between prediction accuracy and feature importance reliability is universal. Here, we demonstrate that, with a sufficient sample size, feature importance (operationalized as Haufe-transformed weights) can achieve fair to excellent split-half reliability. With a sample size of 2600 participants, Haufe-transformed weights achieve average intra-class correlation coefficients of 0.75, 0.57 and 0.53 for cognitive, personality and mental health measures respectively. Haufe-transformed weights are much more reliable than original regression weights and univariate FC-behavior correlations. Original regression weights are not reliable even with 2600 participants. Intriguingly, feature importance reliability is strongly positively correlated with prediction accuracy across phenotypes. Within a particular behavioral domain, there is no clear relationship between prediction performance and feature importance reliability across regression models. Furthermore, we show mathematically that feature importance reliability is necessary, but not sufficient, for low feature importance error. In the case of linear models, lower feature importance error is mathematically related to lower prediction error. Therefore, higher feature importance reliability might yield lower feature importance error and higher prediction accuracy. Finally, we discuss how our theoretical results relate with the reliability of imaging features and behavioral measures. Overall, the current study provides empirical and theoretical insights into the relationship between prediction accuracy and feature importance reliability.