Cograph Regularized Collective Nonnegative Matrix Factorization for Multilabel Image Annotation

Abstract

Automatic image annotation is an effective and straightforward way to facilitate many applications in computer vision. However, manually annotating images is a computation-expensive and labor-intensive task. To address these problems, this paper proposes a novel approach by using a cograph regularized collective nonnegative matrix factorization method to annotate images, which is referred to as CG-CNMF; CG-CNMF maximizes the annotation consistency for each image and minimizes the semantic gap for good annotation performance. To reduce the computation cost, this method formulates the annotation problem as a recommending issue and uses nonnegative matrix factorization (NMF) to recover the image-to-label relation for the testing images. Moreover, to find the most similar latent image features and latent label features during the matrix factorization, it exploits the image-to-image relation and label-to-label relation by utilizing the visual content information of images and the semantic cooccurrence information of labels, respectively. To reduce the semantic gap between the image visual content and semantic concepts, both the semantic features and convolutional neural networks (CNNs)-based visual features are considered. Moreover, to address the label-imbalance and incomplete-label problems, the visual-based label cooccurrence information is also considered. In this way, visually similar images are highly correlated with the true semantics of the test images. The experimental results for three multilabel image datasets demonstrate the effectiveness and the efficiency of the proposed method.

Keywords

• Image annotation
• nonnegative matrix factorization
• collective nonnegative matrix factorization
• semantic gap
• convolutional neural networks