An Agent-Based Data Acquisition Pipeline for Image Data

Abstract

The efficient processing of information plays a critical role in data-driven domains. Data acquisition pipelines, which act as the interface between data collection and subsequent processing, are central to this. Traditional software engineering methods form the foundation for the development of robust data acquisition pipelines, but agent-based systems hold great potential for realizing such pipelines, and this study presents an innovative architecture that uses agent-based components. This modular approach makes it possible to use specialized agents for individual tasks and to increase the effectiveness of the pipelines. The applicability and effectiveness of this architecture are demonstrated using a system for autonomous driving that collects and processes data from edge computers along roads. Our results are on GitHub (https://github.com/BenjaminAcar/Agent-based-Data-Acquisition-Pipeline).

Keywords

• Data pipelines
• agents
• data processing
• streaming
• autonomous driving