Adaptive Resource Allocation and Consolidation for Scientific Workflow Scheduling in Multi-Cloud Environments

Abstract

The emerging multi-cloud environments (MCEs) empower the execution of large-scale scientific workflows (SWs) with sufficient resource provisioning. However, due to complex task dependencies in SWs and various cost-performance of cloud resources, the SW scheduling in MCEs faces huge challenges. To address these challenges, we propose an Online Workflow Scheduling algorithm based on Adaptive resource Allocation and Consolidation (OWS-A2C). In OWS-A2C, the deadline reassignment is first executed for SW tasks based on the execution performance of instance resources, which enhances resource utilization from a local perspective when executing an SW. Next, the execution instances are allocated and consolidated according to the performance requirements of multiple SWs, which improves resource utilization and reduces the total costs of executing multiple SWs from a global perspective. Finally, the SW tasks are dynamically scheduled to the execution instances with the earliest-deadline-first (EDF) discipline and completed before their sub-deadlines. The extensive simulation experiments are conducted to demonstrate the effectiveness of the proposed OWS-A2C on SW scheduling in MCEs, which outperforms three baseline scheduling methods with higher resource utilization and lower execution costs under deadline constraints.

Keywords

• Multi-cloud environments
• scientific workflows
• scheduling optimization
• resource allocation
• resource consolidation