Automatic Classification of Cichlid Behaviors Using 3D Convolutional Residual Networks
Lijiang Long,
Zachary V. Johnson,
Junyu Li,
Tucker J. Lancaster,
Vineeth Aljapur,
Jeffrey T. Streelman,
Patrick T. McGrath
Affiliations
Lijiang Long
School of Biological Sciences, Georgia Institute of Technology, Atlanta, GA 30332, USA; Interdisciplinary Graduate Program in Quantitative Biosciences, Georgia Institute of Technology, Atlanta, GA 30332, USA
Zachary V. Johnson
School of Biological Sciences, Georgia Institute of Technology, Atlanta, GA 30332, USA
Junyu Li
School of Biological Sciences, Georgia Institute of Technology, Atlanta, GA 30332, USA
Tucker J. Lancaster
School of Biological Sciences, Georgia Institute of Technology, Atlanta, GA 30332, USA; Interdisciplinary Graduate Program in Quantitative Biosciences, Georgia Institute of Technology, Atlanta, GA 30332, USA
Vineeth Aljapur
School of Biological Sciences, Georgia Institute of Technology, Atlanta, GA 30332, USA
Jeffrey T. Streelman
School of Biological Sciences, Georgia Institute of Technology, Atlanta, GA 30332, USA; Parker H. Petit Institute of Bioengineering and Bioscience, Georgia Institute of Technology, Atlanta, GA 30332, USA; Corresponding author
Patrick T. McGrath
School of Biological Sciences, Georgia Institute of Technology, Atlanta, GA 30332, USA; Parker H. Petit Institute of Bioengineering and Bioscience, Georgia Institute of Technology, Atlanta, GA 30332, USA; School of Physics, Georgia Institute of Technology, Atlanta, GA 30332, USA; Corresponding author
Summary: Many behaviors that are critical for survival and reproduction are expressed over extended time periods. The ability to inexpensively record and store large volumes of video data creates new opportunities to understand the biological basis of these behaviors and simultaneously creates a need for tools that can automatically quantify behaviors from large video datasets. Here, we demonstrate that 3D Residual Networks can be used to classify an array of complex behaviors in Lake Malawi cichlid fishes. We first apply pixel-based hidden Markov modeling combined with density-based spatiotemporal clustering to identify sand disturbance events. After this, a 3D ResNet, trained on 11,000 manually annotated video clips, accurately (>76%) classifies the sand disturbance events into 10 fish behavior categories, distinguishing between spitting, scooping, fin swipes, and spawning. Furthermore, animal intent can be determined from these clips, as spits and scoops performed during bower construction are classified independently from those during feeding.
