Millisecond‐scale behaviours of plankton quantified in vitro and in situ using the Event‐based Vision Sensor

Abstract

Abstract The Event‐based Vision Sensor (EVS) is a bio‐inspired sensor that captures detailed motions of objects, aiming to become the ‘eyes’ of machines like self‐driving cars. Compared to conventional frame‐based image sensors, the EVS has an extremely fast motion capture equivalent to 10,000‐fps even with standard optical settings, plus high dynamic ranges for brightness and also lower consumption of memory and energy. Here, we developed 22 characteristic features for analysing the motions of aquatic particles from the EVS raw data and tested the applicability of the EVS in analysing plankton behaviour. Laboratory cultures of six species of zooplankton and phytoplankton were observed, confirming species‐specific motion periodicities up to 41 Hz. We applied machine learning to automatically classify particles into four categories of zooplankton and passive particles, achieving an accuracy up to 86%. At the in situ deployment of the EVS at the bottom of Lake Biwa, several particles exhibiting distinct cumulative trajectory with periodicities in their motion (up to 16 Hz) were identified, suggesting that they were living organisms with rhythmic behaviour. We also used the EVS in the deep sea, observing particles with active motion and periodicities over 40 Hz. Our application of the EVS, especially focusing on its millisecond‐scale temporal resolution and wide dynamic range, provides a new avenue to investigate organismal behaviour characterised by rapid and periodical motions. The EVS will likely be applicable in the near future for the automated monitoring of plankton behaviour by edge computing on autonomous floats, as well as quantifying rapid cellular‐level activities under microscopy.

Keywords

• computer vision
• deep sea
• dynamic vision sensor
• event camera
• high‐speed camera
• marine particles