Multimodal Federated Learning in AIoT Systems: Existing Solutions, Applications, and Challenges

Abstract

The unprecedented technological advancements in Artificial Intelligence (AI) and the Internet of Things (IoT) have given rise to ecosystems of intelligent, interconnected devices, forming the Artificial Intelligence of Things (AIoT). These systems, due to security and privacy concerns, necessitate solutions that adhere to the data-driven learning paradigm of Federated Learning (FL) while simultaneously addressing data and resource heterogeneity. This survey, which focuses on smart environments within the transportation, agriculture, manufacturing, and medical sectors, begins with an in-depth review of FL methods and multimodal data-driven learning methodologies. Subsequently, various sensor modalities employed in these environments are presented, as well as the most common multimodal data fusion strategies and their associated fusion operators. The study then explains its shift in focus from data-level to model-level cooperation, delving into multimodal FL (MMFL) systems by categorizing architectures, data processing methods, and model aggregation rules. Emphasis is given to the case of heterogeneous MMFL, as it is identified as the most promising and relatively novel direction that has received insufficient attention due to its recent emergence. The paper introduces a novel classification of the strategies employed when agents have different sensing modalities and model architectures, offering an in-depth analysis of how various fusion approaches can be adapted to accommodate the diversity in data and models. Finally, it examimes the utilization of MMFL in the four application domains and concludes with an analysis of open challenges and future research directions in this promising field.

Keywords

• Aggregation rules
• Artificial Intelligence of Things
• federated learning
• multimodal federated learning
• multimodal fusion