Self‐supervised multimodal fusion transformer for passive activity recognition

Abstract

Abstract The pervasiveness of Wi‐Fi signals provides significant opportunities for human sensing and activity recognition in fields such as healthcare. The sensors most commonly used for passive Wi‐Fi sensing are based on passive Wi‐Fi radar (PWR) and channel state information (CSI) data, however current systems do not effectively exploit the information acquired through multiple sensors to recognise the different activities. In this study, new properties of the Transformer architecture for multimodal sensor fusion are explored. Different signal processing techniques are used to extract multiple image‐based features from PWR and CSI data such as spectrograms, scalograms and Markov transition field (MTF). The Fusion Transformer, an attention‐based model for multimodal and multi‐sensor fusion is first proposed. Experimental results show that the Fusion Transformer approach can achieve competitive results compared to a ResNet architecture but with much fewer resources. To further improve the model, a simple and effective framework for multimodal and multi‐sensor self‐supervised learning (SSL) is proposed. The self‐supervised Fusion Transformer outperforms the baselines, achieving a macro F1‐score of 95.9%. Finally, this study shows how this approach significantly outperforms the others when trained with as little as 1% (2 min) of labelled training data to 20% (40 min) of labelled training data.

Keywords

• deep learning
• multimodal/sensor fusion
• passive WiFi‐based HAR
• self‐supervised learning
• vision transformer (ViT)