Autonomous Discovery and Management in Virtual Container Clusters

Joshua Higgins, Violeta Holmes, Colin Venters

Research output: Contribution to journalArticle

Abstract

Global software stacks on scientific cluster computing resources are required to provide a homogeneous software environment which is typically inflexible. Efforts to integrate Virtual Machines (VMs), in order to abstract the software environment of various scientific applications, suffer from performance limitations and require systems administration expertise to maintain. However, the motivation is clear; in addition to increasing resource utilization, the burden of supporting new software installations on existing systems can be reduced. In this paper, we introduce the Virtual Container Cluster (VCC) that encapsulates a typical HPC software environment within Docker containers. The novel component cluster–watcher enables context aware discovery and configuration of the virtual cluster. Containers offer a lightweight alternative to VMs that more closely match the native performance, and presents a solution that is more accessible to customization by the average user. Combined with a Software Defined Networking (SDN) technology, the VCC enables dynamic features such as transparent scaling and spanning across multiple physical resources. Although SDN introduces an additional performance limitation, within the context of a parallel communication network, the benchmarking demonstrates that this cost is application dependent. The Linpack benchmarking shows that the overhead of container virtualization and SDN interconnect is comparable to the native performance.
Original languageEnglish
Pages (from-to)240-252
Number of pages13
JournalComputer Journal
Volume60
Issue number2
Early online date19 Dec 2016
DOIs
Publication statusPublished - Feb 2017

Fingerprint

Containers
Benchmarking
Cluster computing
Telecommunication networks
Software defined networking
Costs
Virtual machine

Cite this

Higgins, Joshua ; Holmes, Violeta ; Venters, Colin. / Autonomous Discovery and Management in Virtual Container Clusters. In: Computer Journal. 2017 ; Vol. 60, No. 2. pp. 240-252.
@article{8c67ac243cc14d89b2f03e18259d12bb,
title = "Autonomous Discovery and Management in Virtual Container Clusters",
abstract = "Global software stacks on scientific cluster computing resources are required to provide a homogeneous software environment which is typically inflexible. Efforts to integrate Virtual Machines (VMs), in order to abstract the software environment of various scientific applications, suffer from performance limitations and require systems administration expertise to maintain. However, the motivation is clear; in addition to increasing resource utilization, the burden of supporting new software installations on existing systems can be reduced. In this paper, we introduce the Virtual Container Cluster (VCC) that encapsulates a typical HPC software environment within Docker containers. The novel component cluster–watcher enables context aware discovery and configuration of the virtual cluster. Containers offer a lightweight alternative to VMs that more closely match the native performance, and presents a solution that is more accessible to customization by the average user. Combined with a Software Defined Networking (SDN) technology, the VCC enables dynamic features such as transparent scaling and spanning across multiple physical resources. Although SDN introduces an additional performance limitation, within the context of a parallel communication network, the benchmarking demonstrates that this cost is application dependent. The Linpack benchmarking shows that the overhead of container virtualization and SDN interconnect is comparable to the native performance.",
keywords = "cluster computing, Linux containers, Docker, HPC, Software Defined Networking",
author = "Joshua Higgins and Violeta Holmes and Colin Venters",
year = "2017",
month = "2",
doi = "10.1093/comjnl/bxw102",
language = "English",
volume = "60",
pages = "240--252",
journal = "Computer Journal",
issn = "0010-4620",
publisher = "Oxford University Press",
number = "2",

}

Autonomous Discovery and Management in Virtual Container Clusters. / Higgins, Joshua; Holmes, Violeta; Venters, Colin.

In: Computer Journal, Vol. 60, No. 2, 02.2017, p. 240-252.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Autonomous Discovery and Management in Virtual Container Clusters

AU - Higgins, Joshua

AU - Holmes, Violeta

AU - Venters, Colin

PY - 2017/2

Y1 - 2017/2

N2 - Global software stacks on scientific cluster computing resources are required to provide a homogeneous software environment which is typically inflexible. Efforts to integrate Virtual Machines (VMs), in order to abstract the software environment of various scientific applications, suffer from performance limitations and require systems administration expertise to maintain. However, the motivation is clear; in addition to increasing resource utilization, the burden of supporting new software installations on existing systems can be reduced. In this paper, we introduce the Virtual Container Cluster (VCC) that encapsulates a typical HPC software environment within Docker containers. The novel component cluster–watcher enables context aware discovery and configuration of the virtual cluster. Containers offer a lightweight alternative to VMs that more closely match the native performance, and presents a solution that is more accessible to customization by the average user. Combined with a Software Defined Networking (SDN) technology, the VCC enables dynamic features such as transparent scaling and spanning across multiple physical resources. Although SDN introduces an additional performance limitation, within the context of a parallel communication network, the benchmarking demonstrates that this cost is application dependent. The Linpack benchmarking shows that the overhead of container virtualization and SDN interconnect is comparable to the native performance.

AB - Global software stacks on scientific cluster computing resources are required to provide a homogeneous software environment which is typically inflexible. Efforts to integrate Virtual Machines (VMs), in order to abstract the software environment of various scientific applications, suffer from performance limitations and require systems administration expertise to maintain. However, the motivation is clear; in addition to increasing resource utilization, the burden of supporting new software installations on existing systems can be reduced. In this paper, we introduce the Virtual Container Cluster (VCC) that encapsulates a typical HPC software environment within Docker containers. The novel component cluster–watcher enables context aware discovery and configuration of the virtual cluster. Containers offer a lightweight alternative to VMs that more closely match the native performance, and presents a solution that is more accessible to customization by the average user. Combined with a Software Defined Networking (SDN) technology, the VCC enables dynamic features such as transparent scaling and spanning across multiple physical resources. Although SDN introduces an additional performance limitation, within the context of a parallel communication network, the benchmarking demonstrates that this cost is application dependent. The Linpack benchmarking shows that the overhead of container virtualization and SDN interconnect is comparable to the native performance.

KW - cluster computing

KW - Linux containers

KW - Docker

KW - HPC

KW - Software Defined Networking

U2 - 10.1093/comjnl/bxw102

DO - 10.1093/comjnl/bxw102

M3 - Article

VL - 60

SP - 240

EP - 252

JO - Computer Journal

JF - Computer Journal

SN - 0010-4620

IS - 2

ER -