Big Data-Based Improved Data Acquisition and Storage System for Designing Industrial Data Platform

Abstract

Big data-based acquisition and storage system (ASS) plays an important role in the design of industrial data platform. Many big data frameworks have been integrated compression and serialization method. These methods cannot meet the needs of industrial production information management for requiring time-consuming and mass storage. Based on the existing big data frameworks, we propose an enhanced industrial big data platform in order to reduce the data processing time while requiring fewer data storage space. Specifically, this paper focuses on evaluating the impact of multiple compression and serialization methods on big data platform performance and tries to choose optimal compression and serialization methods for the industrial data platform. Compared to the methods integrated in Hadoop and Spark, the experimental results showed the data compression time of the platform has been reduced by 73.9% with a less than 96% the size of data compressed, furthermore, the data serialization time has been reduced by 80.8%. With the increasing amount of data, it takes less time to compare with benchmark methods.

Keywords

• Big data
• acquisition and storage system (ASS)
• industrial data platform
• data acquisition
• data storage
• data compression