A Computer Vision System for an Automated Scoring of a Hand-drawn Geometric Figure

Abstract

Visual Reproduction is a condition to measure Visual Spatial Memory as one of the cognitive domains commonly used to measure visuo-spatial memory. Geometric figures serve as stimulus material, and probands have to reproduce the figures from memory through a hand drawing. The scoring of the drawing has subjective elements. This study aims to evaluate the scoring criteria for the Figural Reproduction Test (FRT), part of the Indonesian Neuropsychological Test Battery, and to develop and evaluate an automated scoring system based on computer vision technology (FRT-CVAS). Scoring evaluation conducted by Cohen Kappa analysis, accuracy, sensitivity, and specificity. The analyzes of the three criteria of the manual confirmed a subjective element in the scoring of the shape of triangles by a moderate (0.74) inter-rater agreement; this agreement could be improved to 0.84 by a slight modification of its criteria. FRT-CVAS, based on computer vision’s identification of the different elements of the hand drawing, was developed and trained using 290 drawings. The system was additionally tested by comparing its scoring with the scoring of two independent raters on 120 drawings from a second data set. FRT-CVAS recognized all elements, and its comparison between human raters showed a high accuracy and sensitivity (minimally 0.91), while the specificity was 0.80 for one of the three criteria. FRT-CVAS offers a highly standardized, consistent, precise, and objective output from the first card in the FRT. This approach is advantageous to data-hungry alternatives such as deep learning when applied to the automated scoring of hand drawings with relatively little data available for training.