Dual-Stream Spatiotemporal Networks with Feature Sharing for Monitoring Animals in the Home Cage

Abstract

This paper presents a spatiotemporal deep learning approach for mouse behavioral classification in the home-cage. Using a series of dual-stream architectures with assorted modifications for optimal performance, we introduce a novel feature sharing approach that jointly processes the streams at regular intervals throughout the network. The dataset in focus is an annotated, publicly available dataset of a singly-housed mouse. We achieved even better classification accuracy by ensembling the best performing models; an Inception-based network and an attention-based network, both of which utilize this feature sharing attribute. Furthermore, we demonstrate through ablation studies that for all models, the feature sharing architectures consistently outperform the conventional dual-stream having standalone streams. In particular, the inception-based architectures showed higher feature sharing gains with their increase in accuracy anywhere between 6.59% and 15.19%. The best-performing models were also further evaluated on other mouse behavioral datasets.

Keywords

• mouse phenotyping
• machine learning
• supervised learning
• video classification
• spatiotemporal