Multi-scale deep learning and clustering-based tabletop object instance segmentation for robot manipulation

Abstract

3D object instance segmentation plays a vital role in various applications such as autonomous driving, robotics and virtual reality. However, tabletop scenes exhibit diverse object complexities and size variations. The challenge is to enhance the accuracy of segmenting these scenes for multiple object instances. This limitation directly impacts robots’ capabilities to effectively grasp and manipulate objects. In this paper, we propose a multi-scale deep learning and clustering-based approach for object instance segmentation in tabletop scenes. Our approach incorporates a multi-scale neighborhood feature sampling (MNFS) module specifically designed to extract local features, and a clustering algorithm to eliminate noise and preserve instance integrity. Furthermore, we combine the strength of both methods through ScoreNet and non-maximal suppression. We conducted extensive experiments on TO-Scene, the first large-scale dataset of 3D tabletop scenes, and observed an average mIoU improvement of approximately 4.07% compared to existing methods. This highlights the superior performance of our proposed method. In addition, we tested our algorithm on a real-scene robotics platform and showed that it has good performance and generalization capabilities to support future applications such as robot grasping.