A multi‐modal clustering method for traditional Chinese medicine clinical data via media convergence

Abstract

Abstract Media convergence is a media change led by technological innovation. Applying media convergence technology to the study of clustering in Chinese medicine can significantly exploit the advantages of media fusion. Obtaining consistent and complementary information among multiple modalities through media convergence can provide technical support for clustering. This article presents an approach based on Media Convergence and Graph convolution Encoder Clustering (MCGEC) for traditional Chinese medicine (TCM) clinical data. It feeds modal information and graph structure from media information into a multi‐modal graph convolution encoder to obtain the media feature representation learnt from multiple modalities. MCGEC captures latent information from various modalities by fusion and optimises the feature representations and network architecture with learnt clustering labels. The experiment is conducted on real‐world multi‐modal TCM clinical data, including information like images and text. MCGEC has improved clustering results compared to the generic single‐modal clustering methods and the current more advanced multi‐modal clustering methods. MCGEC applied to TCM clinical datasets can achieve better results. Integrating multimedia features into clustering algorithms offers significant benefits compared to single‐modal clustering approaches that simply concatenate features from different modalities. It provides practical technical support for multi‐modal clustering in the TCM field incorporating multimedia features.

Keywords

• graph convolutional encoder
• media convergence
• multi‐modal clustering
• traditional Chinese medicine