Extraction of Key-Frames From Endoscopic Videos by Using Depth Information

Abstract

Early detection of colorectal cancer (CRC) can reduce the risk of death. Polyps are the precursor to such cancer. Analyzing the polyps from the most significant frames out of thousands of endoscopy frames is vital for diagnosing and understanding disease. In this article, a deep learning-based monocular depth estimation (MDE) technique is proposed to select the most informative frames (key-frames) of an endoscopic video. In most cases, ground truth depth maps of polyps are not readily available, and that is why the transfer learning approach is adopted in our method. An endoscopic modality generally captures thousands of frames. In this scenario, it is quite essential to discard low-quality and clinically irrelevant frames of an endoscopic video while the most informative frames should be retained for clinical diagnosis. In this view, a key-frame selection strategy is proposed by utilizing the depth information of polyps. In our method, image moment, edge magnitude, and key points are considered for adaptively selecting the key-frames. One important application of our proposed method could be the 3D reconstruction of polyps with the help of extracted key-frames. It gives a surgeon a real-time 3D view of the polyp surface for resection which involves detaching the polyp from its mucosa layer. Also, polyps are localized with the help of extracted depth maps.

Keywords

• Key-frames
• colorectal cancer (CRC)
• monocular depth
• polyps
• 3D reconstruction