A Comprehensive Evaluation of Large Language Models for Turkish Abstractive Dialogue Summarization

Abstract

Text summarization is the task of generating a short and concise summary of a source text. In an abstractive text summarization, the generated summaries may potentially contain new phrases that do not appear in the source text. Dialogue summarization is a special case of text summarization in which the source text is a dialogue between two or more people. Dialogue summarization can be a crucial step especially when the source dialogues are complex and long such as call center conversations. Large language models (LLMs) show remarkable performance in natural language generation tasks and thus they can be a suitable modeling approach for abstractive text summarization. Although LLMs are extensively studied for common languages, there are only a few studies for underrepresented languages such as Turkish. In this paper, we make a comprehensive evaluation of LLMs for Turkish abstractive dialogue summarization. For this purpose, we translated 3 datasets in English to Turkish. Additionally, we make use of a test set that contains real call center dialogues originally collected in Turkish. In the experiments, we observe that fine-tuning LLMs to the dialogue summarization task significantly improves the performance. We obtain 21% overall absolute improvement with the fine-tuning over a baseline Turkish LLM. The performance is improved in all 4 test cases. Additionally, we observe that the length of the summaries plays a crucial role in the performance.

Keywords

• Abstractive dialogue summarization
• large language models
• natural language generation
• text summarization