Scheme of enhancing real-time business processing capabilities based on Kudu for the big data platform

Abstract

Aiming at the difficulty of real-time business processing capability of Hadoop big data platform,the theory of the most advanced Kudu column storage was studied as an effective complement of HDFS block storage,and the use of primary key index and memory acceleration provided by Kudu and Spark was illustrated to effectively solve the big data platform cannot support the technical implementation methods of business pain points such as real-time warehousing,incremental update and SQL-related query.The experimental results prove the effect of the method on improving the real-time business processing capability of the big data platform.

Keywords

• Kudu;big data;column storage;primary key index;memory acceleration;real-time access;incremental update;SQL join query