BAM-SLDK: biologically inspired attention mechanism with spiking learnable delayed kernel synapses

Abstract

Spiking neural networks are emerging as an alternative neural network model due to their biological plausibility, energy efficiency, and built-in ability to learn from temporal dynamics. However, in order to effectively process data with rich spatial and temporal dependencies, the usual static projections (feedforward and recurrent) among layers of spiking neurons fail to represent all the information needed. Inspired by how synaptic delays affect the learning process in biological neurons, in this paper, we propose a biologically inspired attention mechanism based on spiking convolutions with learnable delayed kernel synapses. The proposed model increases temporal learning ability, attending simultaneously to spatial and temporal dynamics with few parameters required. More precisely, our main technical contributions are: (1) we add kernels to the temporal dimension to enlarge the receptive field of the convolution; (2) we time kernels activations to mimic multiple delayed times; and (3) we introduce three different pruning techniques to optimize the number of delays and parameters used. Experiments show that our method surpasses conventional spiking convolutional modules and achieves state-of-the-art results. When pruning, we show that, for some datasets or pruning techniques, removing up to 80% of the initially trained delays results in minimal performance loss, effectively reducing memory consumption and parameters required. To the best of our knowledge, this is the first time that learnable delayed synapses have been included in spiking convolutional layers for neuromorphic datasets classification, unlocking a new biologically inspired attention mechanism and achieving superior performance on high temporal demanding tasks.

Keywords

• spiking neural networks (SNNs)
• delays
• convolutions
• gesture recognition