SWOSBC: A Novel Optimizer for Learning Convolutional Neural Network

Abstract

Deep Neural Networks (DNNs) that aim to maximize accuracy and decrease loss can be trained using optimization algorithms. One of the most significant fields of research is the creation of an efficient optimization technique. Most adaptive optimizers, including Adam and diffGrad, are unable to address noisy updating or zigzag behavior introduced in the optimization process. Moreover, the Adam technique over fits a model in certain situations, especially when the training dataset is small. This ultimately leads to poor generalization efficiency on test data. To get over this shortcoming, an optimization method using the square root of the exponentially weighted average and regulating the step size without applying the second bias correction in addition (SWOSBC) has been proposed that produces second-order moments using both the first and second decay rates instead of only the second decay rate and the second momentum instead of the second bias correction used as the denominator. Using an adaptive term that uses the exponentially weighted average, the suggested SWOSBC offers a smoother trajectory and greater picture Classification Accuracy (CA). The comprehensive tests using standard datasets (CIFAR10, CIFAR100, MNIST, and ImageNet) in comparison to cutting-edge techniques show that SWOSBC performs better. On the CIFAR10 and MNIST datasets for every one of the tested networks, as well as on the CIFAR-100 dataset for the majority of examined network models, the SWOSBC algorithm provides the most effective CA. Using the ImageNet dataset and the ResNet-18 network yields the best classification accuracy. By employing the Rosenbrock functions of convergence and linear regression, it can be observed how smoothly and quickly the SWOSBC reaches the global minima. Source code link: https://github.com/UtpalNandi/SWOSBC.

Keywords

• Adam
• adaptive learning
• classification accuracy
• deep learning
• diffGrad
• neural networks