A Comparative Analysis of Early and Late Fusion for the Multimodal Two-Class Problem

Abstract

In this article we carry out a comparison between early (feature) and late (score) multimodal fusion, for the two-class problem. The comparison is made first from a general perspective, and then from a specific mathematical analysis. Thus, we deduce the error probability expressions for the uncorrelated and correlated multivariate Gaussian distribution, assuming perfect model knowledge (Bayes error rates). We also deduce the corresponding expressions when the model is to be learned from a finite training set, demonstrating its convergence to the Bayes error rates as the training set size goes to infinite. These expressions also demonstrates that early fusion is the best option with model knowledge, and that both early and late fusion degrade due to a finite training set. This degradation is showed to be greater for early fusion due to the dimensionality increase of the feature space, so, eventually, late fusion could be a better option in a practical setting. The mathematical analysis also suggests the convenience of using a, so called, convergence factor, to quantify if a training set size is appropriate for the error probability to be close enough to the Bayes error rate. Different simulated experiments have been made to verify the validity of the mathematical analysis, as well as its possible extension to non-Gaussian models.

Keywords

• Multimodal two-class classification
• early fusion
• late fusion
• probability of error
• training set size