Similarity‐based adversarial knowledge distillation using graph convolutional neural network

Abstract

Abstract This letter presents an adversarial knowledge distillation based on graph convolutional neural network. For knowledge distillation, many methods have been proposed in which the student model individually and independently imitates the output of the teacher model on the input data. Our method suggests the application of a similarity matrix to consider the relationship among output vectors, compared to the other existing approaches. The similarity matrix of the output vectors is calculated and converted into a graph structure, and a generative adversarial network using graph convolutional neural network is applied. We suggest similarity‐based knowledge distillation in which a student model simultaneously imitates both of output vector and similarity matrix of the teacher model. We evaluate our method on ResNet, MobileNet and Wide ResNet using CIFAR‐10 and CIFAR‐100 datasets, and our results outperform results of the baseline model and other existing knowledge distillations like KLD and DML.