Memory management in deep learning:a survey

Abstract

In recent years,deep learning has achieved great success in many fields.As the deep neural network develops towards a deeper and wider direction,the training and inference of a deep neural network face huge memory pressure.The limited memory space of accelerating devices has become an important factor restricting the rapid development of deep neural network.How to achieve efficient memory management in deep learning has become a key point in the development of deep learning.Therefore,the basic characteristics of deep neural network were introduced firstly and memory bottleneck in deep learning training was analyzed.Some representative research works were classified,and their advantages and disadvantages were analyzed.Finally,some important direction and tendency of memory management in deep learning were suggested.

Keywords

• memory management;deep learning;memory swapping;recomputation;memory sharing;compression