A Hybrid Contextual Embedding and Hierarchical Attention for Improving the Performance of Word Sense Disambiguation

Abstract

Word Sense Disambiguation is determining the correct sense of an ambiguous word within context. It plays a crucial role in natural language applications such as machine translation, question-answering, chatbots, information retrieval, sentiment analysis, and overall language comprehension. Recent advancements in this area have focused on utilizing deep contextual models to address these challenges. However, despite this positive progress, semantical and syntactical ambiguity remains a challenge, especially when dealing with polysomy words, and it is considered an AI-complete problem. In this work, we propose an approach that integrates hierarchical attention mechanisms and BERT embeddings to enhance WSD performance. Our model, incorporating local and global attention, demonstrates significant improvements in accuracy, particularly in complex sentence structures. To the best of our knowledge, our model is the first to incorporate hierarchical attention mechanisms integrated with contextual embedding. We conducted experiment on publicly available datasets for English and Italian language. Experimental results show that our model achieves state-of-the-art results in WSD, surpassing baseline models up to 2.9% F1 accuracy on English WSD. Additionally, it demonstrates superior performance in Italian WSD, outperforming existing papers up to 0.7% F1 accuracy. We further adapted the model for Amharic word sense disambiguation. Despite the absence of a standard benchmark dataset for Amharic WSD, our model achieved an accuracy of 92.4% on a dataset we prepared ourselves. Our findings underscore the significance of linguistic features in contextual information capture for WSD. While Part-of-Speech (POS) tagging has a limited impact, word embeddings significantly influence performance. Local and global attention further improve results, particularly at the word level. Overall the results emphasize the importance of context in WSD, advancing context-aware natural language processing systems.

Keywords

• Contextual embeddings
• hierarchical attention
• natural language processing
• word sense disambiguation