Multi‐modal knowledge graph inference via media convergence and logic rule

Abstract

Abstract Media convergence works by processing information from different modalities and applying them to different domains. It is difficult for the conventional knowledge graph to utilise multi‐media features because the introduction of a large amount of information from other modalities reduces the effectiveness of representation learning and makes knowledge graph inference less effective. To address the issue, an inference method based on Media Convergence and Rule‐guided Joint Inference model (MCRJI) has been proposed. The authors not only converge multi‐media features of entities but also introduce logic rules to improve the accuracy and interpretability of link prediction. First, a multi‐headed self‐attention approach is used to obtain the attention of different media features of entities during semantic synthesis. Second, logic rules of different lengths are mined from knowledge graph to learn new entity representations. Finally, knowledge graph inference is performed based on representing entities that converge multi‐media features. Numerous experimental results show that MCRJI outperforms other advanced baselines in using multi‐media features and knowledge graph inference, demonstrating that MCRJI provides an excellent approach for knowledge graph inference with converged multi‐media features.

Keywords

• logic rule
• media convergence
• multi‐modal knowledge graph inference
• representation learning