Research on Knowledge Injection Method for Large Language Model Oriented to Process Specification Texts

Abstract

The application of large language models in process specifications is an effective approach to addressing the issue of inaccurate process knowledge queries. At present, the domain model construction methods through domain knowledge graph embedding or fine-tuning with instruction data are not effective. The difficulty lies in the fact that the process knowledge in the process specifications involves relationships between multiple process elements, which is highly complex. The data are sparse because the standards are only used through citation. The high complexity of process knowledge and sparse data limit the model’s ability to learn process domain concepts, the relationships between concepts and attributes, the relationships between concepts, the relationships between multiple concepts, and reference-based knowledge. To address this difficulty, this paper proposes a large language model knowledge injection method for process specification texts. According to the characteristics of process specification data, this paper designs knowledge injection data including auxiliary sentence identification task, concept-chapter generation task, chapter continuation task and chapter-summary generation task. The model is fine-tuned through supervised learning by combining question-answer pair data to inject domain concepts, attributes, relationships between multiple concepts, and reference knowledge into the model. Experimental results show that the model trained with knowledge injection data and question-answer pair data improves ACC (accuracy) by 7.3 percentage points, ROUGE-L by 7.4 percentage points, and BLEU-4 by 6.2 percentage points compared with the model trained only with question-answer pair data, indicating the effectiveness of the proposed knowledge injection method.

Keywords

• process specification; large language model; knowledge injection; supervised fine-tuning