ChatGPT translation vs. human translation: an examination of a literary text

Abstract

This study evaluates the proficiency of ChatGPT-based translation compared to Human Translation (HT) using an Arabic literary work. It also examines potential translation gaps in ChatGPT and explores its potential to replace human translators. The research analyzes 12 excerpts from Mawsim Al-Hijra Ela Al-Shamal (1966) by Tayeb Salih, comparing the English translation by Denys Johnson-Davies (Season of Migration to the North, 1969) with ChatGPT’s output. A mixed-method approach (qualitative and quantitative) was used, assessing translations through three dimensions of the Multidimensional Quality Metrics (MQM) framework: accuracy, fluency, and design. The MQM scoring model was also employed to ensure reliability. The findings show that HT is more accurate, with an average accuracy score of 94.5%, compared to 77.9% for ChatGPT. However, ChatGPT produces fluent translations, scoring 97.2% in fluency versus 96.6% for HT. Despite its fluency, ChatGPT struggles with design-related elements and often introduces superfluous content. The study concludes that ChatGPT is not a fully reliable tool for translating Arabic literature, which requires professional human translators like Denys Johnson-Davies to ensure accuracy and cultural sensitivity.

Keywords

• ChatGPT
• human translation
• multidimensional quality metrics
• literary translation
• Artificial Intelligence
• Translation & Interpretation