TrapezoidalNet: A new network architecture inspired from the numerical solution of ordinary differential equations

Abstract

The architecture of deep neural networks is commonly determined via trial and error, resulting in inefficiency and a lack of architecture interpretability. Recent research shows that numerical solutions of ordinary differential equations have demonstrated great potential for designing network architectures. Implicit methods are generally more stable and accurate than explicit methods but are yet to be applied to neural network design. Unlike explicit methods that directly calculate the current state based on the previous state, implicit methods have to solve a nonlinear equation to get the approximate solution, which discourages using implicit methods in network design. In this paper, we propose replacing Euler's method with the trapezoidal rule, a scheme that is a second-order implicit method with much lower local truncation errors. The crux of our approach lies in first altering the trapezoidal rule from its implicit form to the explicit form and then designing the corresponding neural network. By introducing the multi-channel convolution and layer-sharing mechanism, we propose a new architecture called TrapezoidalNet. We expect TrapezoidalNet can outperform other deep neural networks based on explicit methods such as ResNet, FitResNet, FractalNet, and LM-ResNet on image recognition tasks. Experimental results on CIFAR10/100 datasets verify the effectiveness of TrapezoidalNet.

Keywords

• Neural network design
• Ordinary differential equations
• Numerical approximation methods
• Image classifications