Advances in Medical Education and Practice (Jun 2020)
Construct Validity of an Instrument for Assessment of Reflective Writing-Based Portfolios of Medical Students
Abstract
Salah Eldin Kassab,1 Mubarak Bidmos,1 Michail Nomikos,1 Suhad Daher-Nashif,2 Tanya Kane,2 Srikant Sarangi,3 Marwan Abu-Hijleh1 1Department of Basic Medical Sciences, College of Medicine, QU Health, Qatar University, Doha, Qatar; 2Department of Population Medicine, College of Medicine, Qatar University, QU Health, Doha, Qatar; 3Danish Institute of Humanities and Medicine (DIHM), Aalborg University, Aalborg, DenmarkCorrespondence: Salah Eldin KassabPhysiology & Medical Education, College of Medicine, QU Health, Qatar University, PO Box 2713, Doha, QatarTel +974 – 4403 7843Email [email protected]: Assessment of reflective writing for medical students is challenging, and there is lack of an available instrument with good psychometric properties. The authors developed a new instrument for assessment of reflective writing-based portfolios and examined the construct validity of this instrument.Methods: After an extensive literature review and pilot testing of the instrument, two raters assessed the reflective writing-based portfolios from years 2 and 3 medical students (n=135) on three occasions. The instrument consists of three criteria: organization, description of an experience and reflection on the experience. We calculated the reliability of scores using generalizability theory with a fully crossed design and two facets (raters and occasions). In addition, we measured criterion validity by testing correlations with students’ scores using other assessment methods.Results: The dependability (Φ) coefficient of the portfolio scores was 0.75 using two raters on three occasions. Students’ portfolio scores represented 46.6% of the total variance across all score comparisons. The variance due to occasions was negligible, while the student–occasion interaction was small. The variance due to student–rater interaction represented 17.7%, and the remaining 27.7% of the variance was due to unexplained sources of error. The decision (D) study suggested that an acceptable dependability (Φ = 0.70 and 0.72) can be achieved by using two raters for one and two occasions, respectively. Finally, we found moderate to large effect-size correlations between students’ scores in reflective writing-based portfolios and communication skills (r = 0.47) and PBL tutorials (r = 0.50).Conclusion: We demonstrated the presence of different sources of evidence that support construct validity of the study instrument. Further studies are warranted before utilizing this instrument for summative assessment of students’ reflective writing-based portfolios in other medical schools.Keywords: G-theory, validity, reflective writing, portfolio, reliability, student assessment
