A Stage-Level Network Parallelization Method Based on Depth Decomposition

Abstract

Parallel computational operations can significantly enhance network computational efficiency, and such processing has a wide range of applications across different spatial scales in networks. However, in the stage-wise level of networks, the majority still defaults to maintaining a serial single-chain structure. We propose a method of splitting across the depth dimension in multiple consecutive stacked computational units, extending such parallel computational operations to the stage-wise level of the network. This type of processing does not introduce excessive computational latency at the network stage level, and the additional computation primarily comes from the Fusion structures between different branches and the widened stage Transition layers. However, due to the relatively small proportion of this additional computational load compared to the entire network and its ease of maintenance, it is manageable. Compared to the performance of a single chain network scaled in depth, the introduction of parallel structures compresses its depth and diversifies the width of the network stage level. Improved the relationship between the initial network’s accuracy and computational efficiency within a certain range. Through subsequent improvements to the stage transition layers and the introduction of branch attention, the performance of the parallelized structure can be further enhanced. In conclusion, our approach provides a viable practice for introducing parallel structures within the stage level of stage-wise networks. By transforming the original serial structure of continuously stacked computational units in the stage level into a parallel structure with multiple subnets, we can achieve superior overall performance compared to the original network within a certain range. The code is available at https://github.com/forrest996/ResNet_P.

Keywords

• Deep learning
• neural networks
• parallelization
• computational efficiency