CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly

Abstract

State-of-the-art methods for cross-modal recipe retrieval failed to consider an underlying but challenging issue, i.e., matching imperfectly problem hidden in positive image-recipe pairs, which is a culprit causing over-fitting. To make up this defect, two critical questions—how to effectively recognize and filter out mismatching parts during the model training and how to pick out and preserve as much matching information as possible need to be answered. To do so, this article proposes a novel method—Cross-modal Recipe rEtrieval by Avoiding Matching imperfectlY, abbreviated as CREAMY, which involving a new-designed learning strategy called Non-Matching and Partial-Matching (NMPM) to undertake two tasks: 1) no longer forcibly aligning each positive image-recipe pair but rather capturing the complementary information from negative pairs; 2) delicately picking up and aligning the matchable part in each pair. To the best of our knowledge, this attempt is a pioneer to defeat the matching imperfectly issue for cross-modal recipe retrieval task. Empirical analysis conducted on Recipe1M dataset validates the advantages of CREAMY over several state-of-the-arts. The code is available at: https://github.com/pouqual/CREAMY.

Keywords

• Cross-modal recipe retrieval
• matching imperfectly
• non-matching
• partial-matching