Are evaluations in simulated medical encounters reliable among rater types? A comparison between standardized patient and outside observer ratings of OSCEs

Easton N. Wollney; Taylor S. Vasquez; Carolyn Stalvey; Julia Close; Merry Jennifer Markham; Lynne E. Meyer; Lou Ann Cooper; Carma L. Bylund

PEC Innovation (Dec 2023)

Are evaluations in simulated medical encounters reliable among rater types? A comparison between standardized patient and outside observer ratings of OSCEs

Easton N. Wollney,
Taylor S. Vasquez,
Carolyn Stalvey,
Julia Close,
Merry Jennifer Markham,
Lynne E. Meyer,
Lou Ann Cooper,
Carma L. Bylund

Affiliations

Easton N. Wollney: Dept. of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA; Corresponding author at: University of Florida, College of Medicine, 2004 Mowry Road #2230-C36, Gainesville, FL 32603, Unites States of America.
Taylor S. Vasquez: College of Journalism and Communications, University of Florida, Gainesville, FL, USA
Carolyn Stalvey: Dept. of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA; College of Journalism and Communications, University of Florida, Gainesville, FL, USA; Dept. of Internal Medicine, College of Medicine, University of Florida, Gainesville, FL USA; Dept. of Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA; Dept. of Hematology and Oncology, College of Medicine, University of Florida, Gainesville, FL, USA; Graduate Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA
Julia Close: Dept. of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA; College of Journalism and Communications, University of Florida, Gainesville, FL, USA; Dept. of Internal Medicine, College of Medicine, University of Florida, Gainesville, FL USA; Dept. of Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA; Dept. of Hematology and Oncology, College of Medicine, University of Florida, Gainesville, FL, USA; Graduate Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA
Merry Jennifer Markham: Dept. of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA; College of Journalism and Communications, University of Florida, Gainesville, FL, USA; Dept. of Internal Medicine, College of Medicine, University of Florida, Gainesville, FL USA; Dept. of Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA; Dept. of Hematology and Oncology, College of Medicine, University of Florida, Gainesville, FL, USA; Graduate Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA
Lynne E. Meyer: Dept. of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA; College of Journalism and Communications, University of Florida, Gainesville, FL, USA; Dept. of Internal Medicine, College of Medicine, University of Florida, Gainesville, FL USA; Dept. of Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA; Dept. of Hematology and Oncology, College of Medicine, University of Florida, Gainesville, FL, USA; Graduate Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA
Lou Ann Cooper: Dept. of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA; College of Journalism and Communications, University of Florida, Gainesville, FL, USA; Dept. of Internal Medicine, College of Medicine, University of Florida, Gainesville, FL USA; Dept. of Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA; Dept. of Hematology and Oncology, College of Medicine, University of Florida, Gainesville, FL, USA; Graduate Medical Education, College of Medicine, University of Florida, Gainesville, FL, USA
Carma L. Bylund: Dept. of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA

Journal volume & issue: Vol. 2
p. 100125

Abstract

Read online

Objective: By analyzing Objective Structured Clinical Examination (OSCE) evaluations of first-year interns’ communication with standardized patients (SP), our study aimed to examine the differences between ratings of SPs and a set of outside observers with training in healthcare communication. Methods: Immediately following completion of OSCEs, SPs evaluated interns’ communication skills using 30 items. Later, two observers independently coded video recordings using the same items. We conducted two-tailed t-tests to examine differences between SP and observers’ ratings. Results: Rater scores differed significantly on 21 items (p < .05), with 20 of the 21 differences due to higher SP in-person evaluation scores. Items most divergent between SPs and observers included items related to empathic communication and nonverbal communication. Conclusion: Differences between SP and observer ratings should be further investigated to determine if additional rater training is needed or if a revised evaluation measure is needed. Educators may benefit from adjusting evaluation criteria to decrease the number of items raters must complete and may do so by encompassing more global questions regarding various criteria. Furthermore, evaluation measures may be strengthened by undergoing reliability and validity testing. Innovation: This study highlights the strengths and limitations to rater types (observers or SPs), as well as evaluation methods (recorded or in-person).

Published in PEC Innovation

ISSN: 2772-6282 (Online)
Publisher: Elsevier
Country of publisher: Netherlands
LCC subjects: Medicine: Public aspects of medicine
Website: https://www.sciencedirect.com/journal/pec-innovation

About the journal

Abstract

Keywords