Spatial Shift Point-Wise Quantization

Abstract

Deep neural networks (DNN) have been applied to numerous artificial-intelligence applications because of their remarkable accuracy. However, computational requirements for deep neural networks are recently skyrocketing far beyond the Moore's Law. In addition to the importance of accuracy, the industry's demand for efficiency in model learning process is increasing. This has led to various attempts to make DNNs more lightweight. Hence, we propose a modeling technique that applies lightweight convolutional neural networks (CNN) to handle the model-learning processes for DNNs. The proposed spatial-shift pointwise quantization (SSPQ) model elegantly combines compact network-design techniques to revitalize DNN quantization efficiency with little accuracy loss. We set the depths of our SSPQ model to 20, 34, and 50 to test against CIFAR10, CIFAR100, and ImageNet datasets, respectively. By applying SSPQ20 to the CIFAR10 dataset, we reduced accuracy degradation by 2.95%, while reducing the number of parameters 8.7×. For the same dataset, our “wide”SSPQ20 variant reduced training parameters 1.96×, compared with the ResNet20 model architecture, which provided a 0.2% improvement. By applying SSPQ34 to the CIFAR100 dataset, we successfully reduced the number of learning parameters 8×, compared with the ResNet34 model with an accuracy degradation of 4.57%. By applying SSPQ50 to the ImageNet dataset, we successfully reduced the number of parameters 10.2× over ResNet50 with an accuracy degradation of only 2.68%. Therefore, using only 9.78MB of learning parameter, the SSPQ50 model guarantees 73.6% accuracy. This is an improved performance by 1.46% in terms of accuracy and 3.62MB in terms of model size compared to the MobileNetV2 lightweight model. The source code is available at https://github.com/EunhuiKim/SSPQ.

Keywords

• Compact neural network design
• quantization
• lightweight modeling
• convolution neural network