Critical appraisal of RCTs by 3rd year undergraduates after short courses in EBM compared to expert appraisal

Abstract

Introduction: An essential aim of courses in evidence-based medicine (EBM) is to improve the skills for reading and interpreting medical literature adequately. Regarding the conceptual framework, it is important to consider different educational levels.Aim: Our primary aim was to investigate the applicability of different instruments for the assessment of methodological study quality by 3rd grade students after short courses in EBM. Our secondary outcomes were agreement with expert assessments and student’s knowledge and competences.Methods: We conducted four short courses in EBM of 90 minutes each for health care management and medical students focused on critical appraisal of the literature. At the end, the students assessed five publications about randomized controlled trials (RCTs) using five different instruments; the results were compared to expert assessments.Results: In total, 167 students participated in our EBM courses. Students’ assessments showed a non-systematic over- and underestimation of risk of bias compared to expert assessments with no clear direction. Agreement with expert assessments ranged between 66% to over 80%. Across RCTs, evidence was found that the choice of instrument had an impact on agreement rates between expert and student assessments (p=0.0158). Three RCTs showed an influence of the instrument on the agreement rate (p<0.05 each). Discussion: Our results contrast sharply with those of many other comparable evaluations. Reasons may be a lack of students’ motivation due to the compulsory courses, and the comparison to a reference standard in addition to self-ratings causing objectivity.Conclusion: Undergraduates should become familiar with the principles of EBM, including research methods, and the reading of scientific papers as soon as possible. For a deeper understanding, clinical experience seems to be an indispensable precondition. Based on our results, we would recommend an integration of lectures about EBM and critical appraisal at least twice during studies and with greater intensity shortly before graduation.

Keywords

• Critical appraisal
• evidence-based medicine
• randomized controlled trial
• training
• undergraduates