Exploiting Big Data for Experiment Reporting: The Hi-Drive Collaborative Research Project Case

Abstract

As timely information about a project’s state is key for management, we developed a data toolchain to support the monitoring of a project’s progress. By extending the Measurify framework, which is dedicated to efficiently building measurement-rich applications on MongoDB, we were able to make the process of setting up the reporting tool just a matter of editing a couple of .json configuration files that specify the names and data format of the project’s progress/performance indicators. Since the quantity of data to be provided at each reporting period is potentially overwhelming, some level of automation in the extraction of the indicator values is essential. To this end, it is important to make sure that most, if not all, of the quantities to be reported can be automatically extracted from the experiment data files actually used in the project. The originating use case for the toolchain is a collaborative research project on driving automation. As data representing the project’s state, 330+ numerical indicators were identified. According to the project’s pre-test experience, the tool is effective in supporting the preparation of periodic progress reports that extensively exploit the actual project data (i.e., obtained from the sensors—real or virtual—deployed for the project). While the presented use case concerns the automotive industry, we have taken care that the design choices (particularly, the definition of the resources exposed by the Application Programming Interfaces, APIs) abstract the requirements, with an aim to guarantee effectiveness in virtually any application context.

Keywords

• big data architecture
• project monitoring and reporting
• non-relational DB
• RESTful APIs
• field operational tests
• automated driving