Multimodal Fusion Classification Network Based on Distance Confidence Score

Abstract

Multimodal data modeling can effectively overcome the problem of insufficient information in a single mode and can greatly improve the performance of model. However, not much progress has been made in quantifying the confidence of neural network models, especially for multimodal fusion models. This paper proposes a method based on embedding, which calculates the local density estimation in the embedding space by calculating the distance between samples, and then calculates the confidence score of the model. The proposed method is scalable and can be used not only for a single modal model, but also for the confidence measurement of multimodal fusion model. In addition, it can also evaluate and quantify the influences of different modal data on the multimodal fusion model.

Keywords

• confidence
• multimodal fusion
• neural network