Nuclear atypia grading in breast cancer histopathological images based on CNN feature extraction and LSTM classification

Abstract

Abstract Early diagnosis of breast cancer, the most common disease among women around the world, increases the chance of treatment and is highly important. Nuclear atypia grading in histopathological images plays an important role in the final diagnosis and grading of breast cancer. Grading images by pathologists is a time consuming and subjective task. Therefore, the existence of a computer‐aided system for nuclear atypia grading is very useful and necessary. In this study, two automatic systems for grading nuclear atypia in breast cancer histopathological images based on deep learning methods are proposed. A patch‐based approach is introduced due to the large size of the histopathological images and restriction of the training data. In the proposed system I, the most important patches in the image are detected first and then a three‐hidden‐layer convolutional neural network (CNN) is designed and trained for feature extraction and to classify the patches individually. The proposed system II is based on a combination of the CNN for feature extraction and a two‐layer Long short‐term memory (LSTM) network for classification. The LSTM network is utilised to consider all patches of an image simultaneously for image grading. The simulation results show the efficiency of the proposed systems for automatic nuclear atypia grading and outperform the current related studies in the literature.

Keywords

• feature extraction
• cancer
• biomedical optical imaging
• medical image processing
• image classification
• convolutional neural nets