Comparison of zero-shot approach and retrieval-augmented generation for analyzing the tone of comments in the Ukrainian language

Abstract

Background. The constant growth of information, online news and text messages in social networks causes new challenges for society. It requires robust tools for analyzing information in real-time, including determining its emotional tone. Understanding the emotional aspect directly affects customer satisfaction in various areas of activity and can suggest directions for improving processes. Therefore, the development of tools for analyzing the tonality of texts can provide the ability to accurately recognize people's emotions, identify problems, and determine ways to solve them. Methods. In this study, approaches to the application of the Mistral-7B-UK large language model were implemented for the text tone analysis. Two datasets of comments in the Ukrainian language were utilized: one for binary classification, divided into negative and positive classes, and another for multiclass classification which included a neutral tonality. These datasets contain reviews about shops, restaurants, hotels, medical facilities, entertainment centers, fitness clubs, the provision of various services, etc. Results and Discussion. The prompts were constructed for the zero-shot approach, describing the role, output format, and additional explanation about tonalities. To implement RAG, Qdrant was utilized as a vector database, while the LangChain library enabled the integration of a large language model with external data sources. To determine text tonality, the five most semantically similar chunks with the defined tonality are retrieved from the vector database, and predefined placeholders are filled in the prompt template. The model's response is generated using the provided context. Conclusion. Research showed that the zero-shot approach achieves higher text tone analysis accuracy than the Retrieval-Augmented Generation model. For binary classification, the overall accuracy was 94 %, and for multiclass – 75 %. The benefit of using external sources was found during the model's recognition of neutral tonality. However, it was observed that comments with opposing tonality could be retrieved as context due to the shared object of description, which negatively affects results.

Keywords

• text tone
• large language model
• zero-shot
• retrieval-augmented generation