Load prediction based elastic resource scheduling strategy in Flink

Abstract

In order to solve the problem that the load of big data stream computing platform fluctuates drastically while the cluster was suffering from the performance bottleneck due to the shortage of computing resources,the load prediction based elastic resource scheduling strategy in Flink (LPERS-Flink) was proposed.Firstly,the load prediction model was set up as the foundation to propose the load prediction algorithm and predict the variation tendency of the processing load.Secondly,the resource judgment model was set up to identify the performance bottleneck and resource redundancy of the cluster while the resource scheduling algorithm was proposed to draw up the resource rescheduling plan.Finally,the online load migration algorithm was proposed to execute the resource rescheduling plan and migrate processing load among nodes efficiently.The experimental results show that the strategy provides better performance promotion in the application with drastically fluctuating processing load.The scale and resource configuration of the cluster responded to the variation of processing load in time and the communication overhead of the load migration was reduced effectively.

Keywords

• stream computing;resource scheduling;load prediction;performance bottleneck;Flink