Deep Motion-Appearance Convolutions for Robust Visual Tracking

Haojie Li; Sihang Wu; Shuangping Huang; Kin-Man Lam; Xiaofen Xing

doi:10.1109/ACCESS.2019.2958405

IEEE Access (Jan 2019)

Deep Motion-Appearance Convolutions for Robust Visual Tracking

Haojie Li,
Sihang Wu,
Shuangping Huang,
Kin-Man Lam,
Xiaofen Xing

Affiliations

Haojie Li: ORCiD; College of Electronic and Information Engineering, South China University of Technology, Guangzhou, China
Sihang Wu: ORCiD; College of Electronic and Information Engineering, South China University of Technology, Guangzhou, China
Shuangping Huang: ORCiD; College of Electronic and Information Engineering, South China University of Technology, Guangzhou, China
Kin-Man Lam: ORCiD; College of Electronic and Information Engineering, Hong Kong Polytechnic University, Hong Kong
Xiaofen Xing: College of Electronic and Information Engineering, South China University of Technology, Guangzhou, China

DOI: https://doi.org/10.1109/ACCESS.2019.2958405
Journal volume & issue: Vol. 7
pp. 180451 – 180466

Abstract

Read online

Visual tracking is a challenging task due to unconstrained appearance variations and dynamic surrounding backgrounds, which basically arise from the complex motion of the target object. Therefore, the information and the correlation between the target motion and its resulting appearance should be considered comprehensively to achieve robust tracking performance. In this paper, we propose a deep neural network for visual tracking, namely the Motion-Appearance Dual (MADual) network, which employs a dual-branch architecture, by using deep two-dimensional (2D) and deep three-dimensional (3D) convolutions to integrate the local and global information of the target object's motion and appearance synchronously. For each frame of a tracking video, 2D convolutional kernels of the deep 2D branch slide over the frame to extract its global spatial-appearance features. Meanwhile, 3D convolutional kernels of the deep 3D branch are used to collaboratively extract the appearance and the associated motion features of the visual target from successive frames. By sliding the 3D convolutional kernels along a video sequence, the model is able to learn the temporal features from previous frames, and therefore, generate the local patch-based motion patterns of the target. Sliding the 2D kernels on a frame and the 3D kernels on a frame cube synchronously enables a better hierarchical motion-appearance integration, and boosts the performance for the visual tracking task. To further improve the tracking precision, an extra ridge-regression model is trained for the tracking process, based not only on the bounding box given in the first frame, but also on its synchro-frame-cube using our proposed Inverse Temporal Training method (ITT). Extensive experiments on popular benchmark datasets, OTB2013, OTB50, OTB2015, UAV123, TC128, VOT2015 and VOT2016, demonstrate that the proposed MADual tracker performs favorably against many state-of-the-art methods.

Published in IEEE Access

ISSN: 2169-3536 (Online)
Publisher: IEEE
Country of publisher: United States
LCC subjects: Technology: Electrical engineering. Electronics. Nuclear engineering
Website: https://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=6287639

About the journal

Abstract

Keywords