Using subjective judgement to determine the validity of a tutorial performance evaluation instrument

Abstract

Evaluating students’ learning performance is dependent on assessment criteria from which valid inferences can be made about student learning. An existing 36-item instrument used to evaluate baccalaureate nursing students’ performance in problem-based learning tutorials was presented to experts in nursing for their subjective judgement of item validity. Quantitative analysis of data sets from experts’ judgements was used to construct a valid measurement scale for evaluating students’ tutorial performance. The objectives of the study were to determine the content validity of items in a tutorial performance evaluation (TPE) instrument and to determine the construct validity of items through paired comparison of main and sub-items in the instrument. Academic experts (n = 8) from two South African universities were selected by means of purposive, maximum variation sampling. Data were collected in three rounds of the Delphi technique, which incorporated the Subjective Judgement Model for paired comparison of instrument items. Experts’ ratings were captured on a visual analogue scale for each item. Relative item weights were determined using paired comparisons. Statistical analysis resulted in ratio scale data, each item being assigned a ratio relative to its weight. It was concluded that quantitative analysis of subjective judgements is useful to determine the construct validity of items through paired comparison of items in a TPE instrument. This article presents the methodological perspectives of subjective judgement to establish instrument validity. Opsomming Die evaluering van studente se leervermoë is afhanklik van die waardebepalingskriteria waarvan geldige afleidings betreffende die student se leerervaring gemaak kan word. ’n Bestaande instrument met 36 items waarmee baccalaureus-verpleegkundestudente se prestasie in die probleemgebaseerde leertutoriale geëvalueer is, is aan kundiges in verpleegkunde gegee vir subjektiewe beoordeling van die geldigheid van die items. ’n Geldige meetinstrument vir die evaluering van studente se tutoriale prestasie is ontwerp deur van die kwantitatiewe ontleding van die datastelle op grond van die kundiges se oordeel gebruik te maak. Die doelwitte van die studie was om die inhoudsgeldigheid van items in ’n evaluerings-instrument van tutoriale prestasie te bepaal en om die konstrukgeldigheid van items te bepaal deur die gepaarde vergelyking van hoof- en sub-items in die instrument. Akademiese kundiges (n = 8) van twee Suid-Afrikaanse universiteite is deur middel van doelgerigte, maksimale variasie-steekproeftrekking geselekteer. Data is deur middel van drie rondtes van die Delphi-tegniek ingesamel, wat die subjektiewe oordeelmodel vir gepaarde vergelyking van die instrumentitems ingesluit het. Die kundiges se beoordeling is op ’n visueleanaloë-skaal vir elke item weergegee. Relatiewe itemgewigte is deur middel van gepaarde vergelyking bepaal. Statistiese ontleding het verhoudingskaaldata tot gevolg gehad, en elke item is van ’n verhouding relatief tot die gewig voorsien. Daar is bevind dat kwantitatiewe ontleding van subjektiewe beoordeling bruikbaar is om die geldigheid van ’n konstruk deur gepaarde vergelyking van items in ’n evalueringsinstrument van tutoriale prestasie te bepaal. Hierdie artikel bied die metodologiese perspektiewe van die subjektiewe beoordeling aan om die geldigheid van die instrument te bepaal.

Keywords

• problem-based learning
• quantitative analysis
• subjective judgement
• tutorial performance
• validity
• nursing