Measuring Short Text Reuse for the Urdu Language

Abstract

Text reuse occurs when one borrows the text (either verbatim or paraphrased) from an earlier written text. A large and increasing amount of digital text is easily and readily available, making it simpler to reuse but difficult to detect. As a result, automatic detection of text reuse has attracted the attention of the research community due to the wide variety of applications associated with it. To develop and evaluate automatic methods for text reuse detection, standard evaluation resources are required. In this paper, we propose one such resource for a significantly under-resourced language-Urdu, which is widely used in day to day communication and has a large digital footprint particularly in the Indian subcontinent. Our proposed Urdu short text reuse corpus contains 2684 short Urdu text pairs, manually labeled as verbatim (496), paraphrased (1329), and independently written (859). In addition, we describe an evaluation of the corpus using various state-of-the-art text reuse detection methods with binary and multi-classification settings and a set of four classifiers. Output results show that character n-gram overlap using J48 classifier outperform other methods for the Urdu short text reuse detection task.

Keywords

• Urdu text reuse detection
• Urdu corpus
• natural language processing