Meshed Context-Aware Beam Search for Image Captioning

Abstract

Beam search is a commonly used algorithm in image captioning to improve the accuracy and robustness of generated captions by finding the optimal word sequence. However, it mainly focuses on the highest-scoring sequence at each step, often overlooking the broader image context, which can lead to suboptimal results. Additionally, beam search tends to select similar words across sequences, causing repetitive and less diverse output. These limitations suggest that, while effective, beam search can be further improved to better capture the richness and variety needed for high-quality captions. To address these issues, this paper presents meshed context-aware beam search (MCBS). In MCBS for image captioning, the generated caption context is dynamically used to influence the image attention mechanism at each decoding step, ensuring that the model focuses on different regions of the image to produce more coherent and contextually appropriate captions. Furthermore, a penalty coefficient is introduced to discourage the generation of repeated words. Through extensive testing and ablation studies across various models, our results show that MCBS significantly enhances overall model performance.

Keywords

• meshed context-aware
• beam search
• decoding strategy