Taskfarm: A Client/Server Framework for Supporting Massive Embarrassingly Parallel Workloads

Abstract

Taskfarm is a client/server framework that can be used to keep track of massive embarrassingly parallel workloads. The system is split up into two packages: (1) a flask server that hands out new tasks via HTTP and (2) a python client that requests and updates tasks. The server stores task progress in a database. This system has been designed to manage a satellite data processing workflow with hundreds of thousands of tasks with variable compute costs. It can be used for any problem that can be solved using a task farm.

Keywords

• open source
• high performance computing
• client/server framework
• task farm
• embarrassingly parallel workload