SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation

Abstract

We investigate the problem of stroke-level sketch segmentation, which is to automatically assign strokes of a given sketch with semantic labels. Solving the problem of sketch segmentation opens the door for fine-grained sketch interpretation, which can benefit many novel sketch-based applications, including sketch recognition and sketch-based image retrieval. In this paper, we propose an approach for multi-class sketch semantic segmentation by considering it as a sequence-to-sequence generation problem. Specifically, an end-to-end learned network SketchSegNet+, built on recurrent neural networks (RNN), is presented to translate a sequence of strokes into a sequence of semantic labels. In addition, a large-scale stroke-level sketch segmentation dataset is constructed for the first time, which is composed of 150K annotated free-hand human sketch selected from QuickDraw. The dataset will be released publicly. The experimental results of stroke-level sketch semantic segmentation on this novel dataset and the SPG dataset demonstrate the effectiveness of our approach.

Keywords

• Stroke-level sketch segmentation
• recurrent neural network