Auto scaling framework, simulator, and algorithms for the META-pipe backend

Abstract

META-pipe is a metagenomic analysis service provided in the ELIXIR distributed life science infrastructure. It provides assembly of sequence data, functional annotation, and taxonomic profiling. The analysis is computationally intensive and it consist of many jobs which have different requirements and varying complexity and execution times It therefore requires an execution environment that can provide a large set of nodes, and elasticity to scale up and down the resources depending on the current resource needs.

We propose an auto scaling framework that automatically scale clusters and schedule jobs on different execution environments. It adds auto scaling to the META-pipe architecture, and provides a simulator that enables the development and comparison of auto scaling algorithms. No earlier solutions provide the needed ability schedule jobs across multiple execution environments and scale their resources. The earlier solutions only provide scaling for a single cloud or cluster.

We designed our framework to support applications that submit jobs for processing, and to support any type of execution environment. The framework consist of of three key components, an estimator, an auto scaling algorithm and the cloud components, that interact with an auto scaling runtime and a simulator through defined interfaces. The framework relies on an external job manager to schedule jobs on the correct execution environments.By implementing these three components based on the requirements of the application, the users can both visualize the changes made by the algorithm and deploy the auto scaling algorithm to a backend system.

To evaluate the simulator we implemented an estimator, three algorithms and a simulated cloud component. The results show that the simulator can accurately simulated different algorithms and simulate the external scheduler. The results from the algorithms show that integrating and deploying an auto scaling algorithm would prove beneficial for the META-pipe application by removing the need for manual execution environment selection and reducing the total duration of the job queues.