Does It Work for Everyone? The Influence of Demographic Variables on Statistical Reliability

Abstract

Recent developments have highlighted the importance of tailored health education efforts. However, little research has explored differential functioning of survey items for diverse populations. This work explores differences in statistical reliability for multiple scales across demographic groups. Understanding such differences is important in health research, given the rapid shifts occurring in global demographics. Study data were collected from eight years of the National College Health Assessment (n = 885,084), a large-scale annual survey of U.S. university students. Meta-analytic reliability generalization was used to compare reliability of two scale measures for multiple demographic groups. In nearly all cases, there were statistically significant differences in reliability across demographic groups. Researchers should consider relative functioning of any scale employed in their work. For certain demographic groups, various scales may not be sufficiently reliable. However, this may be obfuscated in larger samples, containing large numbers of individuals for whom the scale is sufficiently reliable. We suggest a thorough subsets analysis of data to ensure uniform functioning of items prior to use. Just as health interventions should be tailored to populations of interest, so too must research methods and tools.

Keywords

• reliability
• reliability generalization
• health behavior
• psychometrics
• diversity