Real-coded multi-objective genetic algorithm with effective queuing model for efficient job scheduling in heterogeneous Hadoop environment

Abstract

In the recent past, cloud infrastructure has considerably increased its applicability. This has resulted in effective Big Data processing. Hadoop schedulers are critical components to deliver required levels of efficiency, where MapReduce tasks are assigned to Hadoop nodes by the scheduler. There is a significant challenge in planning the increasing amount of functions and resources in a scalable way. Also, this challenge is further compounded by the potential heterogeneity of the deployed Hadoop. This paper proposes a scheduler that makes scheduling a choice by assessing the entire task group in the job queue. Further, the proposed scheduler uses a new scheduling method based on Real Coded Genetic Algorithm (RCGA). RCGA with MapReduce enables users to create more scalable applications. A higher abstraction is provided in less time. The experimental results indicate that the proposed RCGA scheduler achieves better performance than the existing systems for the following metrics: Execution Time, Total Cost, Resource Utilization, Speedup, Throughput, Scheduling Efficiency, Fairness Relaxation, Scheduling Time, Turnaround Time, CPU Time, Data Locality and Average Node-Locality Ratio.

Keywords

• Hadoop framework
• Heterogeneous Hadoop framework
• Big Data
• MapReduce
• Genetic algorithm