Iterative Back Projection Network Based on Deformable 3D Convolution

Abstract

Video super-resolution technology enhances the display quality of videos by obtaining high-resolution videos from low-resolution videos. Unlike single-image super-resolution, utilizing information between adjacent video frames is crucial in video super-resolution. To improve the performance of video super-resolution reconstruction, a model combining deformable 3D convolution and iterative back projection is proposed to fully exploit the temporal-spatial correlation of video frames. The model takes multiple consecutive video frames as input and outputs the super-resolution reconstruction of the middle frame, including three modules: multi-scale feature extraction, feature fusion, and high-resolution reconstruction. Firstly, multi-scale 3D convolution is used for preliminary feature extraction. Then, deformable 3D convolution and iterative back projection are combined for feature fusion. Finally, multiple residual dense blocks and sub-pixel convolution are used for high-resolution reconstruction, and global residual connections are utilized to obtain the reconstructed high-resolution video. Experimental results on the Vid4 dataset demonstrate that compared to existing methods, this method can effectively improve the peak signal-to-noise ratio and structural similarity performance and achieve better visual effects with 4x super-resolution magnification.

Keywords

• Video super-resolution
• adjacent video frames
• deformable 3D convolution
• iterative back-projection
• multi-scale feature extraction