Exploring region features in remote sensing image captioning

Abstract

Remote sensing image captioning (RSIC), an emerging field of cross-modal tasks, has become a popular research topic in recent years. Feature extraction underlies all RSIC tasks, with current tasks using grid features. Compared with grid features, region features provide object-level location-related information; however, these features have not been considered in the RSIC tasks. Therefore, this study examined the performance of region features on RSIC tasks. We generated region annotations based on published RSIC datasets to address the need for region-related datasets. We extracted region features according to the labeled data and proposed a Region Attention Transformer model. To solve the information loss problem owing to the region of interest pooling during region feature extraction, we proposed region-grid features and used geometry relationships for estimating correlations between different region features. We compared the performances of the models using grid and region features. The results showed that region features performed well in RSIC tasks, and region features forced the model to pay more attention to object regions when generating object-related words. This study describes a novel method of using features in RSIC tasks. Our region annotations are available at https://github.com/zk-1019/exploring.

Keywords

• Transformer model
• Image processing
• Model training
• Remote sensing image captioning