Non‐local neural networks combined with local importance‐based pooling for space‐time video super‐resolution

Abstract

Abstract Compared with convolutional operation, non‐local operation can directly capture long‐range dependencies and thus has a larger receptive field. However, the computation and memory consumption of non‐local operation is much higher than convolutional operation, so it cannot be used repeatedly as a general component directly. In this paper, in order to balance the accuracy and computational complexity of non‐local enhancement, the non‐local operation is simplified based on local importance‐based pooling, which can dynamically extract discriminative features during the down‐sampling process by learning adaptive weights. Such simplified non‐local enhancement is able to prevent unacceptable computational consumption caused by directly processing the entire feature maps containing a large number of features. In order to verify the effectiveness of the proposed method, 2D and 3D feature extraction blocks are constructed based on the simplified non‐local operations, and they are stacked as feature extraction networks for space‐time video super‐resolution task, which aims to increase resolution in both time and space simultaneously. Extensive experiments demonstrate that the proposed simplified non‐local networks can effectively improve the performance of space‐time video super‐resolution task both quantitatively and qualitatively.