Hole-Filling of RealSense Depth Images Using a Color Edge Map

Abstract

The RealSense camera frequently includes hole regions-unfilled regions where depth information are missing in the depth image. The holes cause serious problems in applications that use depth sensors and often straddle the background and object simultaneously. Accordingly, a hole must be filled by considering the boundary of the object. Fortunately, the object boundary inside the hole can be estimated by the RealSense color image. This paper proposes a hole-filling method in which filling converges to the object boundary from both border pixels. However, the color edge elements of the object boundary are often missing. Because the missing edge highly influences the hole-filling procedure, the boundary of the object close to the camera should be filled as completely as possible. Consequently, we introduce a simple method of filling the gap between two edge endpoints. Furthermore, we can address the situation in which there are one or more edge pixels between two border pixels. We repeatedly change the filling direction by alternating between the horizontal and vertical directions until no hole remains. Our experiment reveals that the proposed method outperforms other filling methods when numerically comparing the hole-filled depth image with the ground truth image for four methods: root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity index map (SSIM), and correlation coefficient (CC). The proposed method also produces a superior visual when we observe the 3D stereo image pair constructed using an original color image and a left-view color image generated with depth-image-based rendering (DIBR).

Keywords

• Hole-filling method
• inpainting
• Intel RealSense camera D435
• depth image
• multiview generation