A Levenshtein distance-based method for word segmentation in corpus augmentation of geoscience texts
Jinqu Zhang,
Lang Qian,
Shu Wang,
Yunqiang Zhu,
Zhenji Gao,
Hailong Yu,
Weirong Li
Affiliations
Jinqu Zhang
School of Computer Science, South China Normal University, Guangzhou, China
Lang Qian
School of Computer Science, South China Normal University, Guangzhou, China
Shu Wang
State Key Laboratory of Resources and Environmental Information System, Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing, China
Yunqiang Zhu
State Key Laboratory of Resources and Environmental Information System, Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing, China
Zhenji Gao
Technology Innovation Center of Geological Information of Ministry of Natural Resources, China Geological Survey, Beijing, China
Hailong Yu
Technology Innovation Center of Geological Information of Ministry of Natural Resources, China Geological Survey, Beijing, China
Weirong Li
State Key Laboratory of Resources and Environmental Information System, Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing, China
ABSTRACTFor geoscience text, rich domain corpora have become the basis of improving the model performance in word segmentation. However, the lack of domain-specific corpus with annotation labelled has become a major obstacle to professional information mining in geoscience fields. In this paper, we propose a corpus augmentation method based on Levenshtein distance. According to the technique, a geoscience dictionary of 20,137 words was collected and constructed by crawling the keywords from published papers in China National Knowledge Infrastructure (CNKI). The dictionary was further used as the main source of synonyms to enrich the geoscience corpus according to the Levenshtein distance between words. Finally, a Chinese word segmentation model combining the BERT, Bi-gated recurrent neural network (Bi-GRU), and conditional random fields (CRF) was implemented. Geoscience corpus composed of complex long specific vocabularies has been selected to test the proposed word segmentation framework. CNN-LSTM, Bi-LSTM-CRF, and Bi-GRU-CRF models were all selected to evaluate the effects of Levenshtein data augmentation technique. Experiments results prove that the proposed methods achieve a significant performance improvement of more than 10%. It has great potential for natural languages processing tasks like named entity recognition and relation extraction.
