Can Fuzzy Relational Calculus Bring Complex Issues in Selection of Examiners into Focus?

Abstract

The examinee and the examiner play pivotal roles in the educational grading system. Students’ academic performance evaluation by multiple experts involves epistemic uncertainty, which can be modeled using a fuzzy set theory. How many evaluators/experts are almost similar in their perceptual subjective evaluation of the students answer paper? In other words, how many experts are reliable for a particular evaluation task with a defined possibility level? In this paper, the focus is on object’s features (students’ marks) as a basis in the subjective evaluation process to identify the degree of similarity among the domain experts. The case study reveals that 11 out of 20 evaluators are similar in their decision making of students’ academic performance with possibility (α-level cut, 0.98). The inter-rater reliability (κ-coefficient) among the selected 11 teachers is 0.41, which signifies a fair/moderate agreement in the evaluation process. This paper proposes an approach that is useful for the selection of experts having similar perceptions in judgment. This paper demonstrates a case study showing how it is useful to educational policy makers in the selection of examiners.

Keywords

• education grading system
• examinee and examiner
• inter-rater reliability
• κ-coefficient
• expert’s perception
• fuzzy sets
• fuzzy relational calculus
• cosine amplitude method