Tensor Network Methods for Hyperparameter Optimization and Compression of Convolutional Neural Networks

Abstract

Neural networks have become a cornerstone of computer vision applications, with tasks ranging from image classification to object detection. However, challenges such as hyperparameter optimization (HPO) and model compression remain critical for improving performance and deploying models on resource-constrained devices. In this work, we address these challenges using Tensor Network-based methods. For HPO, we propose and evaluate the TetraOpt algorithm against various optimization algorithms. These evaluations were conducted on subsets of the NATS-Bench dataset, including CIFAR-10, CIFAR-100, and ImageNet subsets. TetraOpt consistently demonstrated superior performance, effectively exploring the global optimization space and identifying configurations with higher accuracies. For model compression, we introduce a novel iterative method that combines CP, SVD, and Tucker tensor decompositions. Applied to ResNet-18 and ResNet-152, we evaluated our method on the CIFAR-10 and Tiny ImageNet datasets. Our method achieved compression ratios of up to 14.5× for ResNet-18 and 2.5× for ResNet-152. Additionally, the inference time for processing an image on a CPU remained largely unaffected, demonstrating the practicality of the method.

Keywords

• tensortrain optimisation
• hyperparameter optimisation
• image classification
• computer vision and pattern recognition
• convolutional neural networks
• model compression