Frontiers in Education (Oct 2024)
Development, field testing, and initial validation of an adherence rubric for caregiver coaching
Abstract
IntroductionThe translation and use of evidence-based practices (EBPs) within early intervention (EI) systems presents challenges. The Office of Special Education (OSEP) has emphasized results-driven accountability to expand state accountability from compliance to also include quality services that align with EBPs. OSEP’s results-driven accountability provided states the opportunity to design State Systemic Improvement Plans (SSIP) to strengthen the quality of EI services by increasing the capacity of EI systems to implement, scale-up, and sustain use of EBPs. Caregiver coaching is widely accepted as an EBP within EI settings, yet uptake and fidelity to coaching practices remains limited. Such widespread implementation of caregiver coaching is partially limited by a lack of measurement tools that operationalize behaviors consistent with coaching. In this study, we describe the development of the Kentucky Coaching Adherence Rubric-Revised (KCAR-R) and psychometric testing of the instrument.MethodsWe developed and tested the KCAR-R to measure fidelity of coaching practices within a state-wide professional development program, the Coaching in Early Intervention Training and Mentorship Program. We define operational elements of the KCAR-R and rubric design elements related to: creators; users and uses; specificity, judgment complexity; evaluative criteria; quality levels; quality definitions; scoring strategies; presentation; explanation; quality processes; accompanying feedback information; secrecy; and exemplars. With regard to psychometric validation of the KCAR-R, interrater reliability was analyzed using intraclass correlation coefficients across eight raters and 301 randomly selected video submissions. Internal consistency was evaluated using Cronbach’s alpha across 429 video submissions.ResultsResults showed 0.987 agreement, indicating excellent interrater reliability; item level internal consistency values ranged from 0.860 to 0.882. for scale if item deleted and 0.834 for the total scale.DiscussionFindings from this study showed that the KCAR-R operationalized behaviors that exemplify caregiver coaching and may serve as a resource for other states or programs to document the quality and fidelity of evidence-based EI services. To influence EI provider practices at a systems level, we used implementation science to guide our work and provide examples of how EI systems seeking ways to create sustainable quality services may build upon our approach.
Keywords
