Is the rating result reliable? A new approach to respond to a medical trainee's concerns about the reliability of Mini-CEX assessment

Abstract

Purpose: Whether the rating result of mini-clinical evaluation exercise (Mini-CEX) for rating clinical skills is reliable is of a medical trainee's great concerns. The objectives of this study were to analyze the test-retest reliability, interrater reliability and internal consistency reliability of Mini-CEX. Methods: Three clinical scenarios, each played by a standardized patient and resident, were developed and videotaped. A group of assessors were recruited to rate the resident's clinical skills using Mini-CEX with a nine-point grading scale in each videotaped clinical scenario. Each assessor was required: (1) to watch the videotaped clinical scenarios a sequential order; (2) to rate each medical trainee's clinical skills in each clinical scenario for two rating sessions, and there must be a minimum three-week interval between the first and the second Mini-CEX rating session. Results: A total of 38 assessors participated in this study. This study showed that: (1) an assessor carried out similar rating reuslts under the same clinical performance based on an acceptable test-retest reliability (Pearson's correlation coefficients = 0.24–0.76, P value=<0.01–0.14); (2) assessors gave similar rating results to a medical trainee's clinical performance based on a good interrater reliability (intra-class correlation coefficient = 0.57–0.83, P value=<0.01–0.03); and (3) the items reflected unidimensionally a construct—a medical trainee's clinical skills based on an excellent internal consistency reliability (Cronbach's alpha = 0.92–0.97). Conclusion: This study convincingly showed that Mini-CEX is a reliable assessment tool for rating clinical skills, and can be widely used to assess medical trainees’ clinical skills.

Keywords

• Workplace-based assessment
• Mini-CEX
• Reliability