Large-scale Assessments in Education (Jun 2022)

Generating group-level scores under response accuracy-time conditional dependence

  • Hyo Jeong Shin,
  • Paul A. Jewsbury,
  • Peter W. van Rijn

DOI
https://doi.org/10.1186/s40536-022-00122-y
Journal volume & issue
Vol. 10, no. 1
pp. 1 – 19

Abstract

Read online

Abstract The present paper investigates and examines the conditional dependencies between cognitive responses (RA; Response Accuracy) and process data, in particular, response times (RT) in large-scale educational assessments. Using two prominent large-scale assessments, NAEP and PISA, we examined the RA-RT conditional dependencies within each item in the measurement model and the structural model. Evidence for RA-RT conditional dependencies was observed in data from both programs, presenting a challenge in incorporating RT to the current operational models in NAEP and PISA that do not account for RA-RT conditional dependencies. However, inclusion of RT in the model had a relatively large contribution to improving the measurement of ability (residual variance decrease of 11% in NAEP and 18% in PISA), in contrast to relatively modest difference in parameter estimation from neglecting the conditional dependencies (e.g., estimated difference on residual variance of 1% in both NAEP and PISA). We conclude that the benefits of incorporating RT in the operational models for large-scale educational assessments may outweigh the costs.

Keywords