Multi-Frame Depth Super-Resolution for ToF Sensor With Total Variation Regularized L1 Function

Abstract

In this paper, we propose a multi-frame depth super-resolution (SR) method based on L1 data fidelity with the total variation regularization (TV-L1) model. The majority of time-of-flight (ToF) sensors exhibit limited spatial resolution compared to RGB sensors and the improvement of the depth image resolution is an inherently ill-posed problem. To overcome this under-determined problem, the solution space is limited by the regularization term through prior knowledge and the data fidelity term using statistical information of the noise. Firstly, the statistical characteristics of ToF depth images are analyzed to specify the appropriate observation model. Thereafter, the objective function for multi-frame depth SR based on the TV-L1 model is designed by considering the prior knowledge of the depth images. This approach enables the sharpness of the edges to be preserved and the noise amplification to be suppressed simultaneously. Furthermore, an efficient solver based on half-quadratic splitting is proposed. The algorithm minimizes the objective function for the multi-frame SR problem consisting of the TV regularization term and L1 data fidelity term. The proposed method is verified on a synthetic dataset and real-world data acquired from a ToF sensor. The experimental results demonstrate that the proposed method can substantially reconstruct high-resolution depth images in terms of preserving sharp depth discontinuities, without any obvious artifacts, and can increase robustness to noise.

Keywords

• Depth super-resolution
• time-of-flight sensor
• multi-frame super-resolution
• total variation regularization
• L1 data fidelity