Adaptive Hard Parameter Sharing Method Based on Multi-Task Deep Learning

Abstract

Multi-task learning (MTL) improves the performance achieved on each task by exploiting the relevant information between tasks. At present, most of the mainstream deep MTL models are based on hard parameter sharing mechanisms, which can reduce the risk of model overfitting. However, negative knowledge transfer may occur, which hinders the performance improvement achieved for each task. In this paper, for situations when multiple tasks are jointly trained, we propose the adaptive hard parameter sharing method. On the basis of the adaptive hard parameter sharing method, the number of nodes in the network is dynamically updated by setting a continuous gradient difference-based sign threshold and a warm-up training iteration threshold through the relationships between the parameters and the loss function. After each task fully utilizes the shared information, adaptive nodes are used to further optimize each task, reducing the impact of negative migration. By using simulation studies and instance analyses, we demonstrate theoretical proof that the performance of the proposed method is better than that of the competing method.

Keywords

• multi-task learning
• continuous gradient difference threshold
• warm-up
• training iteration threshold
• information sharing
• adaptive nodes