Fully Bayesian analysis of the relevance vector machine classification for imbalanced data problem

Abstract

Abstract Relevance Vector Machine (RVM) is a supervised learning algorithm extended from Support Vector Machine based on the Bayesian sparsity model. Relevance Vector Machine classification suffers from theoretical limitations and computational inefficiency mainly because there is no closed‐form solution for the posterior of the weight parameters. We propose two advanced Bayesian approaches for RVM classification, namely the Enhanced RVM and the Reinforced RVM, to perfect the theoretic framework of RVM and extend the algorithm to the imbalanced data problem, which has an arresting skew in data size between classes. First, the Enhanced RVM conducts a strict Bayesian sampling process instead of the approximation method in the original one to remedy its theoretic limitations, especially the non‐convergence of the iterations. Secondly, we conjecture that the hierarchical prior makes the Reinforced RVM achieve consistent estimations of the quantities of interest compared with the non‐consistent estimations of the original RVM. Consistency is necessary for RVM classification since it makes the model more stable and localises the relevant vectors more accurately in the imbalanced data problem. The two‐level prior also renders the Reinforced one competitive in the imbalanced data problem by building the inner connection of parameter dimensions and alloting a more vital relevance to the small class data weight parameter. The theoretic proofs and several numeric studies demonstrate the merits of our two proposed algorithms.