Mutual Learning Knowledge Distillation Based on Multi-stage Multi-generative Adversarial Network

Abstract

Aiming at the problems of insufficient knowledge distillation efficiency,single stage training methods,complex training processes and difficult convergence of traditional knowledge distillation methods in image classification tasks,this paper designs a mutual learning knowledge distillation based on multi-stage multi-generative adversarial networks(MS-MGANs).Firstly,the whole training process is divided into several stages,teacher models of different stages are obtained to guide student models to achieve better accuracy.Secondly,the layer-wise greedy strategy is introduced to replace the traditional end-to-end training mode,and the layer-wise training strategy based on convolution block is adopted to reduce the number of parameters to be optimized in each iteration process,and further improve the distillation efficiency of the model.Finally,a generative adversarial structure is introduced into the knowledge distillation framework,with the teacher model as the feature discriminator and the student model as the feature generator,so that the student model can better follow or even surpass the performance of the teacher model in the process of continuously imitating the teacher model.The proposed method is compared with other advanced knowledge distillation methods on several public image classification data sets,and the experimental results show that the new knowledge distillation method has better performance in image classification.

Keywords

• mutual learning knowledge distillation|layer-wise greedy strategy|generative adversarial network|model compression|image classification