Estimating the number of principal components via Split-Half Eigenvector Matching (SHEM)

Abstract

Estimating the number of principal components to retain for dimension reduction is a critical step in many applications of principal component analysis. Common methods may not be optimal, however. The current paper presents an alternative procedure that aims to recover the true number of principal components, in the sense of the number of independent vectors involved in the generation of the data. • Data are split into random halves repeatedly. • For each split, the eigenvectors in one half are compared to those in the other. • The split between high and low similarities is used to estimate the number of principal components.The method is a proof of principle that similarity over split-halves of the data may provide a useful approach to estimating the number of components in dimension reduction, or of similar dimensions in other models.

Keywords

• SHEM: Split-Half Eigenvector Matching