International Journal of Assessment Tools in Education (Jul 2019)
Investigating a new method for standardising essay marking using levels-based mark schemes
Abstract
Standardisationis a procedure used by Awarding Organisations to maximise marking reliability,by teaching examiners to consistently judge scripts using a mark scheme.However, research shows that people are better at comparing two objects thanjudging each object individually. Consequently, Oxford, Cambridge and RSA (OCR,a UK awarding organisation) proposed investigating a new procedure, involvingranking essays, where essay quality is judged in comparison to other essays.This study investigated the marking reliability yielded by traditionalstandardisation and ranking standardisation. The study entailed a markingexperiment followed by examiners completing a questionnaire. In the controlcondition live procedures were emulated as authentically as possible withinthe confines of a study. The experimental condition involved ranking the quality ofessays from the best to the worst and then assigning marks. After eachstandardisation procedure the examiners marked 50 essays from an AS Historyunit. All participants experienced both procedures, and marking reliability wasmeasured. Additionally, the participants’ questionnaire responses were analysedto gain an insight into examiners’ experience. It is concluded that the RankingProcedure is unsuitable for use in public examinations in its current form. TheTraditional Procedure produced statistically significantly more reliablemarking, whilst the Ranking Procedure involved a complex decision-makingprocess. However, the Ranking Procedure produced slightly more reliable markingat the extremities of the mark range, where previous research has shown thatmarking tends to be less reliable.
Keywords
