MedEdPORTAL (Oct 2007)
Educational Measurement Workshop: A “Sweet Approach” to Understanding the Basic Principles of Educational Measurement
Abstract
Abstract Increasingly, medical educators are expected to design, implement, and analyze the results of an array of learner assessment strategies ranging from multiple-choice examinations to human and mechanical simulations. However, many faculty have received only limited training focused on the core principles of educational measurement and learner assessment. These principles, including reliability, validity, and sources of errors, are presented in a 2.5- to 3-hour workshop using chocolate as a central focus to aid in participant understanding. During the workshop, participants identify key attributes of excellence in chocolate, develop rating scales, train raters, rate the chocolates, and then report their scores to examine rater consistency. While participants sample the chocolates, instructors identify sources of variability by three of the four sources of error used to ground the workshop: instrumentation (clarify of rating scales), raters (fatigue, bias towards a particular chocolate, contamination with another food/drink source), administration (clarity of instructions, order of tasting, chocolate sample size), and any other sources that can introduce error in the measurement. At the conclusion of the workshop, the basic principles of educational measurement and the methods of controlling common sources of error (e.g., instrumentation) are discussed in the context of common learner assessment methods. This workshop has been presented to three sets of primary care physician faculty development participants at the Medical College of Wisconsin in 2000, 2003, and 2006 and at the 2001 Society of Teachers of Family Medicine (STFM) Annual Conference in Denver, CO. In all cases, participants were actively engaged throughout the workshop and rated the experience with consistently high evaluation scores and comments. STFM attendees who submitted evaluations unanimously rated the workshop a 5 (highest) on a 1-5 point scale on all evaluation questions. Comments included “outstanding review of concepts,” “enlightening and concrete way to demonstrate complex ideas,” “excellent way to make people engaged,” “overall, outstanding,” and “well thought out session, one of the best.”
Keywords
