PACL: Piecewise Arc Cotangent Decay Learning Rate for Deep Neural Network Training

Abstract

Deep neural networks (DNNs) are currently the best-performing method for many classification problems. For training DNNs, the learning rate is the most important hyper-parameter, choice of which affects the performance of the model greatly. In recent years, some learning rate schedulers, such as HTD, CLR, and SGDR, have been proposed. These methods, some of which make use of the cycling mechanism to improve the convergence speed and accuracy of DNN, but performance degradation occurs in the convergence process. Others have good accuracy, but their convergence speed is too slow. This paper proposed a new learning rate schedule called piecewise arc cotangent decay learning rate (PACL), which can not only improve the convergence speed and accuracy of DNN but also significantly reduce performance degradation zone caused by the cycling mechanism. It is easy to implement, but almost at no extra computing expense. Finally, we demonstrate the effectiveness of PACL, on training CIFAR-10, CIFAR-100, and Tiny ImageNet with ResNet, DenseNet, WRN, SEResNet, and MobileNet.

Keywords

• Deep neural networks
• learning rate schedulers
• arc cotangent
• optimization