Cosine similarity-guided knowledge distillation for robust object detectors

Abstract

Abstract This paper presents a Cosine Similarity-Based Knowledge Distillation (CSKD) for robust, lightweight object detectors. Knowledge Distillation (KD) has been effective in enhancing the performance of compact models in image classification by leveraging deep CNN models. However, the complex and multifaceted nature of object detection, characterized by its modular design and multitasking requirements, poses significant challenges for traditional KD techniques. These challenges are further compounded by the conventional reliance on the Mean Squared Error (MSE) loss function and the limited application of enhanced feature representations to the training phase. Addressing these limitations, the proposed CSKD method combines cosine similarity guidance with MSE loss to facilitate a more effective knowledge transfer from the teacher model to the student model. This is achieved by distilling both intermediate features and prediction outputs, aided by an assistant prediction branch designed to learn directly from the teacher’s predictions. This dual-faceted distillation strategy enables the student model to better mimic the teacher model’s behavior, leading to improved performance. The proposed method demonstrates versatility and robustness across various object detector architectures without the need for additional feature enhancement layers during training. Notably, employing ResNet-50 as the teacher model and ResNet-18 as the student model, we achieve new benchmarks in KD for object detection across several architectures, including Faster-RCNN, RetinaNet, FCOS, and GFL, with respective mAP scores of 36.6, 35.2, 35.9, and 38.9. These results highlights the effectiveness of CSKD in advancing the state-of-the-art in KD for object detection, offering a compelling solution to the challenges previously faced by traditional KD methods in this domain. The code of the proposed CSKD is available at https://github.com/swkdn16/CSKD .