Calibrated Parsing Items Evaluation: a step towards objectifying the translation assessment

Abstract

Abstract The present research paper introduces a translation evaluation method called Calibrated Parsing Items Evaluation (CPIE hereafter). This evaluation method maximizes translators’ performance through identifying the parsing items with an optimal p-docimology and d-index (item discrimination). This method checks all the possible parses (annotations) in a source text by means of the Brat Visualization Stanford CoreNLP software. CPIE takes a step towards the objectification of translation assessment by allowing evaluators to assess values (impacts) of the items in source texts via docimologically justified parsing items. For this paper, 16 evaluators were recruited to score translation drafts by means of the holistic, analytic, Preselected Items Evaluation (PIE) methods and CPIE method. For the present research paper, “F-Statistics,” “Probability Plot,” “Spearman rho,” and “Regression Variable Plot” were applied to the evaluators’ translation assessments to ensure the degree of validity and reliability of CPIE compared to the holistic, analytic, and PIE methods, respectively. The results indicated that the CPIE method was more consistent and valid in terms of docimologically justified parsing items. The limitations and the possibilities of the CPIE method in web-based platforms were also discussed.

Keywords

• Calibrated parsing items evaluation
• P-docimology
• Item discrimination
• Brat visualization Stanford CoreNLP
• Validity
• Reliability