Complexity-Reduced Super Resolution for Foveation-Based Driving Head Mounted Displays

Abstract

In this paper, we propose a foveation-based super resolution (SR) algorithm to create high resolution images from low resolution inputs for virtual reality head mounted displays. Because the proposed SR algorithm is integrated in the previous foveation-based driving technology to cover the small area around the foveation point that requires high rendering quality, the overall computational complexity is substantially reduced, compared to the whole area SR. The target display has 4 times as high resolution as the input image, therefore, the proposed SR algorithm generates $4\times $ as well as $2\times $ SR images at the same time. To support two SR output images, small area SR, and small number of weights, we employ cropping as well as progressive and recursive framework used in the previous MS-LapSRN. We reduce the number of neurons by placing the deconvolutional layer after convolutional layers, compared to the MS-LapSRN. PSNR and SSIM performances of the proposed SR for the $4\times $ scale are estimated as 31.152 dB and 0.935 for Set5, 26.656 dB and 0.858 for Set14, 27.138 dB and 0.830 for BSD100, and 25.078 dB and 0.836 for Urban100. For the target 8K display of $7,680\times 4.320$ , the proposed FovSR-integrated driving technology achieves the substantial reductions by 76.7 % and 99.02 % on the number of lines from 7,680 to 1,788 and the number of neurons from 24,518,246,400 to 239,541,184, respectively.

Keywords

• Super resolution
• low computational complexity
• virtual reality
• head mounted display
• foveation-based driving