Spectrum Concentration in Deep Residual Learning: A Free Probability Approach

Abstract

We revisit the weight initialization of deep residual networks (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with the limiting spectral distribution of non-Hermitian random matrices, rather than their conventional Hermitian counterparts in the literature. This new tool enables us to evaluate the singular value spectrum of the input-output Jacobian of a fully connected deep ResNet in both linear and nonlinear cases. With the powerful tool of free probability, we conduct an asymptotic analysis of the (limiting) spectrum on the single-layer case, and then extend this analysis to the multi-layer case of an arbitrary number of layers. The asymptotic analysis illustrates the necessity and university of rescaling the classical random initialization by the number of residual units L, so that the squared singular value of the associated Jacobian remains of order O(1), when compared with the large width and depth of the network. We empirically demonstrate that the proposed initialization scheme learns at a speed of orders of magnitudes faster than the classical ones, and thus attests a strong practical relevance of this investigation.

Keywords

• Residual network
• weight initialization
• random matrix theory
• non-Hermitian free probability theory
• Jacobian matrix
• spectral density