Inner Loop-Based Modified Differentiable Architecture Search

Abstract

Differentiable neural architecture search, which significantly reduces the computational cost of architecture search by several orders of magnitude, has become a popular research issue in recent years. Architecture search can fundamentally be described as an optimization problem. The differentiable architecture search updates the search process based on gradients, then derives the final sub-network architecture from the super network of the search space. However, the gap between the super network and its sub-networks together with the inaccuracy of the gradient approximation during architecture optimization bring performance collapse problems in the architecture search, making the search process extremely unstable. To this end, we propose an inner loop-based modified differentiable neural architecture search method (InLM-NAS). Firstly, we redefine the objective function of the architecture optimization process in the search process by introducing an inner-loop mechanism to prevent overfitting problems of architecture parameters and avoid convergence of the architecture search to suboptimal architectures. Secondly, a novel approximation calculation is introduced in the architecture optimization process, which reduces the error caused by the gradient approximation. It alleviates the sensitivity to the hyper-parameters setting during the architecture search and enhances the stability of the architecture search. Finally, extensive validation experiments on public datasets demonstrate that our proposed method has a more robust search process, and the searched neural network architecture has a superior network performance.

Keywords

• Neural network
• differentiable architecture search
• deep learning
• implicit regularization