Orchestrating Docker containers in the HPC environment

Joshua Higgins, Violeta Holmes, Colin Venters

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

36 Citations (Scopus)

Abstract

Linux container technology has more than proved itself useful in cloud computing as a lightweight alternative to virtualisation, whilst still offering good enough resource isolation. Docker is emerging as a popular runtime for managing Linux containers, providing both management tools and a simple file format. Research into the performance of containers compared to traditional Virtual Machines and bare metal shows that containers can achieve near native speeds in processing, memory and network throughput. A technology born in the cloud, it is making inroads into scientific computing both as a format for sharing experimental applications and as a paradigm for cloud based execution. However, it has unexplored uses in traditional cluster and grid computing. It provides a run time environment in which there is an opportunity for typical cluster and parallel applications to execute at native speeds, whilst being bundled with their own specific (or legacy) library versions and support software. This offers a solution to the Achilles heel of cluster and grid computing that requires the user to hold intimate knowledge of the local software infrastructure. Using Docker brings us a step closer to more effective job and resource management within the cluster by providing both a common definition format and a repeatable execution environment. In this paper we present the results of our work in deploying Docker containers in the cluster environment and an evaluation of its suitability as a runtime for high performance parallel execution. Our findings suggest that containers can be used to tailor the run time environment for an MPI application without compromising performance, and would provide better Quality of Service for users of scientific computing.
Original languageEnglish
Title of host publicationHigh Performance Computing
Subtitle of host publication30th International Conference, ISC High Performance 2015, Frankfurt, Germany, July 12-16, 2015, Proceedings
EditorsJulian M. Kunkel, Thomas Ludwig
PublisherSpringer, Cham
Pages506-513
Number of pages8
Volume9137
ISBN (Electronic)9783319201191
ISBN (Print)9783319201184
DOIs
Publication statusPublished - 29 Jun 2015
Event30th International Conference, ISC High Performance 2015 - Frankfurt, Germany
Duration: 12 Jul 201516 Jul 2015
Conference number: 30

Publication series

NameLecture Notes in Computer Science
PublisherSpringer
Volume9137
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference30th International Conference, ISC High Performance 2015
Country/TerritoryGermany
CityFrankfurt
Period12/07/1516/07/15

Cite this