Task-Aware Dual Prototypical Network for Few-Shot Human-Object Interaction Recognition

Abstract

Recognizing human-object interaction (HOI) is an important research topic in computer vision. With the great success of deep learning in image classification, the HOI recognition task has also made great progress. However, the problems of instance imbalance and combinatorial explosion still remain the key challenges, which restrict the performance of HOI recognition methods. Therefore, this paper formulates HOI recognition in a few-shot scene to tackle the above problems and proposes a novel task-aware dual prototypical network (TDP-Net) to address few-shot HOI task. Specifically, it first assigns semantic-aware task representations for different tasks as their prior knowledge, subsequently generates attention weights by semantic graph attention module (SGA-Module). It effectively weights the importance on different regions of the visual features, adaptively for different task conditions, which realizes to reason for novel tasks. In addition, it designs a dual prototypes module (DP-Module) to generate both action class prototypes and object class prototypes, which classifies the verb and noun labels respectively. The complex visual relationships between actions and objects can be effectively separated by constructing class prototypes for actions and objects. Meanwhile, owing to the similarity among the related interactions, the knowledge is transferred to the new interactions by reorganizing the action and object prototypes. The experimental results show that the average accuracies of this model outperform the baseline by 3.2 percentage points and 15.7 percentage points on two exper-imental settings, which verifies its effectiveness on the few-shot HOI task.

Keywords

• computer vision
• image classification
• human-object interaction (hoi)
• few-shot learning (fsl)
• atten-tion mechanism