Fractal and Fractional (Aug 2023)

The Improved Stochastic Fractional Order Gradient Descent Algorithm

  • Yang Yang,
  • Lipo Mo,
  • Yusen Hu,
  • Fei Long

DOI
https://doi.org/10.3390/fractalfract7080631
Journal volume & issue
Vol. 7, no. 8
p. 631

Abstract

Read online

This paper mainly proposes some improved stochastic gradient descent (SGD) algorithms with a fractional order gradient for the online optimization problem. For three scenarios, including standard learning rate, adaptive gradient learning rate, and momentum learning rate, three new SGD algorithms are designed combining a fractional order gradient and it is shown that the corresponding regret functions are convergent at a sub-linear rate. Then we discuss the impact of the fractional order on the convergence and monotonicity and prove that the better performance can be obtained by adjusting the order of the fractional gradient. Finally, several practical examples are given to verify the superiority and validity of the proposed algorithm.

Keywords