Clinical Interventions in Aging (Nov 2023)
External Validation of Models for Predicting Disability in Community-Dwelling Older People in the Netherlands: A Comparative Study
Abstract
Tjeerd van der Ploeg,1 René Schalk,2– 4 Robbert J J Gobbens1,2,5,6 1Faculty of Health, Sports and Social Work, Inholland University of Applied Sciences, Amsterdam, the Netherlands; 2Tranzo, Tilburg University, Tilburg, the Netherlands; 3Human Resource Studies, Tilburg University, Tilburg, the Netherlands; 4Economic and Management Science, North West University, Potchefstroom, South Africa; 5Zonnehuisgroep Amstelland, Amstelveen, the Netherlands; 6Department Family Medicine and Population Health, Faculty of Medicine and Health Sciences, University of Antwerp, Antwerp, BelgiumCorrespondence: Tjeerd van der Ploeg, Inholland University of Applied Sciences, Faculty of Health, Sports and Social Work, De Boelelaan 1109, Amsterdam, 1081 HV, the Netherlands, Tel +31 6 53519264, Email [email protected]: Advanced statistical modeling techniques may help predict health outcomes. However, it is not the case that these modeling techniques always outperform traditional techniques such as regression techniques. In this study, external validation was carried out for five modeling strategies for the prediction of the disability of community-dwelling older people in the Netherlands.Methods: We analyzed data from five studies consisting of community-dwelling older people in the Netherlands. For the prediction of the total disability score as measured with the Groningen Activity Restriction Scale (GARS), we used fourteen predictors as measured with the Tilburg Frailty Indicator (TFI). Both the TFI and the GARS are self-report questionnaires. For the modeling, five statistical modeling techniques were evaluated: general linear model (GLM), support vector machine (SVM), neural net (NN), recursive partitioning (RP), and random forest (RF). Each model was developed on one of the five data sets and then applied to each of the four remaining data sets. We assessed the performance of the models with calibration characteristics, the correlation coefficient, and the root of the mean squared error.Results: The models GLM, SVM, RP, and RF showed satisfactory performance characteristics when validated on the validation data sets. All models showed poor performance characteristics for the deviating data set both for development and validation due to the deviating baseline characteristics compared to those of the other data sets.Conclusion: The performance of four models (GLM, SVM, RP, RF) on the development data sets was satisfactory. This was also the case for the validation data sets, except when these models were developed on the deviating data set. The NN models showed a much worse performance on the validation data sets than on the development data sets.Keywords: prediction models, modeling techniques, external validation, performance, calibration, correlation coefficient, root of the mean squared error
