Hadoop-based secure storage solution for big data in cloud computing environment

Abstract

In order to address the problems of the single encryption algorithm, such as low encryption efficiency and unreliable metadata for static data storage of big data platforms in the cloud computing environment, we propose a Hadoop based big data secure storage scheme. Firstly, in order to disperse the NameNode service from a single server to multiple servers, we combine HDFS federation and HDFS high-availability mechanisms, and use the Zookeeper distributed coordination mechanism to coordinate each node to achieve dual-channel storage. Then, we improve the ECC encryption algorithm for the encryption of ordinary data, and adopt a homomorphic encryption algorithm to encrypt data that needs to be calculated. To accelerate the encryption, we adopt the dual-thread encryption mode. Finally, the HDFS control module is designed to combine the encryption algorithm with the storage model. Experimental results show that the proposed solution solves the problem of a single point of failure of metadata, performs well in terms of metadata reliability, and can realize the fault tolerance of the server. The improved encryption algorithm integrates the dual-channel storage mode, and the encryption storage efficiency improves by 27.6% on average.

Keywords

• Big data security
• Data encryption
• Hadoop
• Parallel encrypted storage
• Zookeeper