A review of Khmer word segmentation and part-of-speech tagging and an experimental study using bidirectional long short-term memory

Abstract

Large contiguous blocks of unsegmented Khmer words can cause major problems for natural language processing applications such as machine translation, speech synthesis, information extraction, etc. Thus, word segmentation and part-of- speech tagging are two important prior tasks. Since the Khmer language does not always use explicit separators to split words, the definition of words is not a natural concept. Hence, tokenization and part-of-speech tagging of these languages are inseparable because the definition and principle of one task unavoidably affect the other. In this study, different approaches using in Khmer word segmentation and part-of-speech are reviewed and experimental study using a single long short-term memory network is described. Dataset from Asia Language Treebank is used to train and test the model. The preliminary experimental model achieved 95% accuracy rate. However, more testing to evaluate the model and compare it with different models is needed to conduct to select the more higher accuracy model.

Keywords

• word segmentation, part-of-speech tagging, khmer natural language processing, lstm