Image Inpainting with Bilateral Convolution

Abstract

Due to sensor malfunctions and poor atmospheric conditions, remote sensing images often miss important information/pixels, which affects downstream tasks, therefore requiring reconstruction. Current image reconstruction methods use deep convolutional neural networks to improve inpainting performances as they have a powerful modeling capability. However, deep convolutional networks learn different features with the same group of convolutional kernels, which restricts their ability to handle diverse image corruptions and often results in color discrepancy and blurriness in the recovered images. To mitigate this problem, in this paper, we propose an operator called Bilateral Convolution (BC) to adaptively preserve and propagate information from known regions to missing data regions. On the basis of vanilla convolution, the BC dynamically propagates more confident features, which weights the input features of a patch according to their spatial location and feature value. Furthermore, to capture different range dependencies, we designed a Multi-range Window Attention (MWA) module, in which the input feature is divided into multiple sizes of non-overlapped patches for several heads, and then these feature patches are processed by the window self-attention. With BC and MWA, we designed a bilateral convolution network for image inpainting. We conducted experiments on remote sensing datasets and several typical image inpainting datasets to verify the effectiveness and generalization of our network. The results show that our network adaptively captures features between known and unknown regions, generates appropriate content for various corrupted images, and has a competitive performance compared with state-of-the-art methods.

Keywords

• image inpainting
• bilateral convolution
• multi-range window attention
• remote sensing images