Custom App Container Targets HPC Needs
Custom versions of application containers are emerging that meet stringent requirements. Among the first is an HPC-specific container platform dubbed "Singularity" built around what its developers say is the concept of "mobility of compute."
Despite the fact that commercial and open source application containers are making steady headway in production settings, researchers overseeing HPC operations at Lawrence Berkeley National Laboratory concluded that Docker and other container approaches are "non-starters."
Hence, the Energy Department lab that is also part of the University of California system developed the open source Singularity approach to address HPC requirements for distributing applications. "With Singularity you can build executable containers based on your host system and define what happens when that container is launched," explained project lead Gregory Kurtzer of Berkeley Labs.
Singularity is designed to handle a range of processes inside the container, including single or a complex of binaries, scripts and data, Kurtzer added.
The approach also combines packaging and container concepts "while maintaining standard command line interface work flows," Berkeley Labs said.
Container-based application portability and distribution run up against "pitfalls" in HPC settings, researchers noted. Among these hurdles is "privilege escalation" within traditional containers that leads to, among other things, network access issues. "Application portability is not necessarily a good use of these technologies," developers noted. Hence, Singularity was developed to address specific HPC requirements.
Kurtzer said the 1.0 release supports:
- The ability to create Singularity containers based on a package specfile.
- Specfile templates that can be generated automatically.
- Automatic dependency resolution, including: dynamic libraries, Perl, Python and R scripts and modules; Basic X11; and Open MPI.
- Direct execution of Singularity containers.
- Access to files in either a home or scratch directory.
- Existing IO all maintained through the container.
- Singularity internal container cache management.
- Standard networking access (as it does on the host).
- Singularity containers running within existing resource contexts.
- Easy integration into existing schedulers and batch scripts.
- Scalable execution of MPI parallel jobs.
Singularity containers also were designed to be portable between Linux distributions, Kurtzer added.
Security has been an ongoing concern as container technology ramps up to handle enterprise workloads. Security concerns in HPC settings were "greatly mitigated," developers noted, by running Singularity containers "100 percent from user space" so that container contents are executed as the run-time user. That approach eliminates the need to configure networks as virtual LANs isolated from other users and file systems.
Stressing portability, Singularity's architects also said the HPC container application itself is a package bundling all required dependencies needed to run the container. Hence, for example, a binary can be distributed among systems without having to worry about compatibility issues.
An introductory Singularity video is available here.
