Journal Home > Volume 22 , Issue 6
Tsinghua Science and Technology 2017, 22(6): 675-681 https://doi.org/10.23919/TST.2017.8195349
Open Access | Issue | Published: 14 December 2017

SCStore: Managing Scientific Computing Packages for Hybrid System with Containers

Show Author's Information Wusheng Zhang,  Jiao Lin,  Weiping Xu,  Haohuan Fu,  Guangwen Yang( )
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
National Supercomputing Center in Wuxi, Wuxi 210008, China.
Department of Earth System Science, Tsinghua University, Beijing 100084, China.
Keywords:
container, hybrid system, high performance computing, package management
Cite this article:
Zhang W, Lin J, Xu W, et al. SCStore: Managing Scientific Computing Packages for Hybrid System with Containers. Tsinghua Science and Technology, 2017, 22(6): 675-681. https://doi.org/10.23919/TST.2017.8195349
Download citation
EndNote(RIS)
BibTeX

511

Views

30

Downloads

Citations

41

Crossref

N/A

WoS

44

Scopus

0

CSCD

Abstract Full text About this article

Managing software packages in a scientific computing environment is a challenging task, especially in the case of heterogeneous systems. It is error prone when installing and updating software packages in a sophisticated computing environment. Testing and performance evaluation in an on-the-fly manner is also a troublesome task for a production system. In this paper, we discuss a package management scheme based on containers. The newly developed method can ease the maintenance complexity and reduce human mistakes. We can benefit from the self-containing and isolation features of container technologies for maintaining the software packages among intricately connected clusters. By deploying the SuperComputing application Strore (SCStore) over the WAN connected world-largest clusters, it proved that it can greatly reduce the effort for maintaining the consistency of software environment and bring benefit to achieve automation.


menu
Abstract
Full text
Outline
About this article

SCStore: Managing Scientific Computing Packages for Hybrid System with Containers

Show Author's information Wusheng Zhang, Jiao Lin, Weiping Xu, Haohuan Fu, Guangwen Yang( )
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
National Supercomputing Center in Wuxi, Wuxi 210008, China.
Department of Earth System Science, Tsinghua University, Beijing 100084, China.

Abstract

Managing software packages in a scientific computing environment is a challenging task, especially in the case of heterogeneous systems. It is error prone when installing and updating software packages in a sophisticated computing environment. Testing and performance evaluation in an on-the-fly manner is also a troublesome task for a production system. In this paper, we discuss a package management scheme based on containers. The newly developed method can ease the maintenance complexity and reduce human mistakes. We can benefit from the self-containing and isolation features of container technologies for maintaining the software packages among intricately connected clusters. By deploying the SuperComputing application Strore (SCStore) over the WAN connected world-largest clusters, it proved that it can greatly reduce the effort for maintaining the consistency of software environment and bring benefit to achieve automation.

Keywords: container, hybrid system, high performance computing, package management

References(16)

[1]
Vallee G., Naughton T., Bohm S., and Engelmann C., A runtime environment for supporting research in resilient HPC system software & tools, in International Symposium on Computing and Networking (CANDAR), 2013, pp. 213-219.
DOI
[2]
Morari A. and Valero M., HPC system software for regular and irregular parallel applications, in IEEE International Symposium on Parallel & Distributed Processing Workshops and PhF Forum, 2013.
DOI
[3]
Wang K., Kulkarni A., Lang M., Arnold D., and Raicu I., Exploring the design tradeoffs for extreme-scale high-performance computing system software, IEEE Transactions on Parallel and Distributed Systems, vol. 27, no. 4, pp. 1070-1084, 2016.
DOI Google Scholar
[4]
Adufu T., Choi J., and Kim Y., Is container-based technology a winner for high performance scientific applications? in Network Operations and Management Symposium (APNOMS), 2015, pp. 507-510.
DOI
[5]
Lv K., Zhao Z., Rao R., Hong P., and Zhang X., PCCTE: A portable component conformance test environment based on container cloud for avionics software development, in International Conference on Progress in Informatics and Computing (PIC), 2017, pp. 664-668.
DOI
[6]
Gamblin T., LeGendre M., Collette M. R., Lee G. L., Moody A., de Supinski B. R., and Futral S., The spack package manager: Bringing order to HPC software chaos, in SC’15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysisy, 2015, pp. 1-12.
DOI
[7]
Becker G., Scheibel P., Gendre M. L., and Gamblin T., Managing combinatorial software installations with spack, international workshop on Hpc user support tools, in SC’15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysisy, 2016, pp. 14-23.
DOI
[8]
Dolstra E., de Jonge M., and Visser E., Nix: A safe and policy-free system for software deployment, in Proceedings of the 18th Large Installation System Administration Conference (LISA XVIII) LISA’04, 2004, pp. 79-92.
[9]
Hoste K., Timmerman J., Georges A., and De Weirdt S., Nix: A safe and policy-free system for software deployment, in High Performance Computing Networking Storage and Analysis Proceedings, 2012, pp. 572-582.
[10]
Yang G., Zhao W., Ding N., and Dun F., “Sunway· Taihulight” and its applications, KEXUE, vol. 69, no. 3, pp.12-16, 2017.
Google Scholar
[11]
DiGirolamo A., The smithy software installation tool, http://github.com/AnthonyDiGirolamo/smithy, 2012.
[12]
HashDist: Reproducible, Relocatable, Customizable, Cross-Platform Software Stacks for Open Hydrological Science, 2013, http://github.com/hashdist.
[13]
Howell M., Homebrew the Missing Package Manager for OS X, http://brew.sh.com, 2017.
[14]
ROCKS: Open Source Toolkit for Real and Virtual Clusters, http://www.rocksclusters.org, 2017.
[15]
The MacPorts Project Official Homepage, http://www.macports.org, 2017.
[16]
About FreeBSD Ports, http://www.freebsd.org/ports, 2017.
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 18 October 2017
Accepted: 16 November 2017
Published: 14 December 2017
Issue date: December 2017

Copyright

© The author(s) 2017

Acknowledgements

Thanks to Prof. Shimin Hu, who gives a lot of valuable comments and advises upon preparing this paper. This work was supported by the National Key R&D Program of China (No. 2016YFA0602100), the National Natural Science Foundation of China (No. 91530323), and Open Fund of Key Laboratory of Data Analysis and Applications, SOA (No. LDAA-2014-03).

Rights and permissions

Return