System performance optimization practice for big data scenarios

Abstract

In the existing large-scale distributed environments, there is still much room for improvement in the performance and computational efficiency of big data applications.However, performance analysis and optimization in large-scale environments requires a large number of human resources from domain experts.This paper proposes a general lowperformance query statement detection and optimization process for performance optimization in big data applications, summarizes four types of low-performance behaviors that significantly affect the performance of big data applications, and proposes specific optimization strategies for each type of low-performance behavior.Finally, through experimental evaluation, the effectiveness of the optimization scheme in actual large-scale cluster is verified.

Keywords

• Hadoop;big data system;performance optimization;tuning tool