Compression artifact reduction of low bit‐rate videos via deep neural networks using self‐similarity prior

Abstract

Abstract This paper presents a hybrid scheme that integratedly uses self‐similarity prior and deep convolutional neural network (CNN) fusion for compression artifact reduction in low bit‐rate video applications. Based on the temporal correlation hypothesis, the self‐similarity prior is extended to the temporal domain by using as references not only the current decoded frame but also its neighbouring frames. Furthermore, being cognizant of that the bicubic downsampling process can typically improve the perceptual quality of a video coded at low bit‐rate, for each small patch in the current frame, we search for similar patches in down‐scaled versions of these references, and then form several self‐similarity prior based predictions by tiling these similar patches at corresponding positions. To further exploit information flow across scales, a deep CNN model is constructed that contains two sub‐networks to estimate the final output. One sub‐network takes the self‐similarity prior based predictions along with the decoded frame itself; and the other takes the down‐scaled versions of these frames as network input. Experimental results demonstrate that the proposed method can remarkably improve, both subjectively and objectively, quality of compressed video sequences of low bit‐rates.