Situation Awareness in AI-Based Technologies and Multimodal Systems: Architectures, Challenges and Applications

Abstract

Situation Awareness (SA) is a process of sensing, understanding and predicting the environment and is an important component in complex systems. The reception of information from the environment tends to be continuous and of a multimodal nature. AI technologies provide a more efficient and robust support by subdividing the different stages of SA objectives into tasks such as data fusion, representation, classification, and prediction. This paper provides an overview of AI and multimodal methods used to build, enhance and evaluate SA in a variety of environments and applications. Emphasis is placed on enhancing perceptual integrity and persistence. Research indicates that the integration of artificial intelligence and multimodal approaches has significantly enhanced perception and comprehension in complex systems. However, there remains a research gap in projecting future situations and effectively fusing multimodal information. This paper summarizes some of the use cases and lessons learned where AI and multimodal techniques have been used to deliver SA. Future perspectives and challenges are proposed, including more comprehensive predictions, greater interpretability, and more advanced visual information.

Keywords

• Artificial intelligence
• situation awareness
• deep learning
• machine learning
• reinforcement learning
• multimodal fusion