Predicting visual difference maps for computer‐generated images by integrating human visual system model and deep learning

Abstract

Abstract The quality of images generated by computer graphics rendering algorithms is mainly affected by visible distortion at some pixel locations. Image quality assessment (IQA) metrics are commonly utilized to assess the quality of rendered images, but their results are a global difference value, which does not provide pixel‐wise differences to optimize the renderings. In contrast, visibility difference models including visual perception models and deep learning models can calculate pixel‐wise visibility difference between distorted images and reference images. However, they either are only applied to a single type of visible distortion or are seriously dependent on datasets. To this end, the authors propose a novel model, dubbed Human Visual Perception and Deep Learning Image Difference Metric (HPDL‐IDM), which combines the Human Visual System (HVS) model and deep learning. HPDL‐IDM primarily consists of two modules: (i) the visual perception feature calculation module, which calculates difference maps between various kinds of features extracted from the reference image and the distorted image according to the visual characteristics of human eyes and concatenates them, and (ii) the deep learning module, which utilizes a neural network of encoder–decoder structure to train on the LocvisVC and VisTexRes datasets whose input and output are these concatenated feature difference maps and the final image distortion visibility difference map respectively. Additionally, the authors pool the final difference map into a global difference value between 0 and 1 to apply their model to many image processing tasks related to Image quality metrics (IQMs). Experimental results show that HPDL‐IDM's generalization capacity and accuracy are improved by a large margin compared to other models.

Keywords

• distortion visibility
• human visual perception
• image quality
• visual metric