AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Big Data Mining and Analytics Article
PDF (6.1 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

Distributed Heterogeneous Spiking Neural Network Simulator Using Sunway Accelerators

Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China, and Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen 518055, China
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
Guangdong Institute of Intelligence Science and Technology, Zhuhai 519031, China
Show Author Information

Abstract

Spiking Neural Network (SNN) simulation is very important for studying brain function and validating the hypotheses for neuroscience, and it can also be used in artificial intelligence. Recently, GPU-based simulators have been developed to support the real-time simulation of SNN. However, these simulators’ simulating performance and scale are severely limited, due to the random memory access pattern and the global communication between devices. Therefore, we propose an efficient distributed heterogeneous SNN simulator based on the Sunway accelerators (including SW26010 and SW26010pro), named SWsnn, which supports accurate simulation with small time step (1/16 ms), randomly delay sizes for synapses, and larger scale network computing. Compared with existing GPUs, the Local Dynamic Memory (LDM) (similar to cache) in Sunway is much bigger (4 MB or 16 MB in each core group). To improve the simulation performance, we redesign the network data storage structure and the synaptic plasticity flow to make most random accesses occur in LDM. SWsnn hides Message Passing Interface (MPI)-related operations to reduce communication costs by separating SNN general workflow. Besides, SWsnn relies on parallel Compute Processing Elements (CPEs) rather than serial Manage Processing Element (MPE) to control the communicating buffers, using Register-Level Communication (RLC) and Direct Memory Access (DMA). In addition, SWsnn is further optimized using vectorization and DMA hiding techniques. Experimental results show that SWsnn runs 1.4−2.2 times faster than state-of-the-art GPU-based SNN simulator GPU-enhanced Neuronal Networks (GeNN), and supports much larger scale real-time simulation.

Keywords

Spiking Neural Network (SNN) simulationSunway acceleratorrandom accessMessage Passing Interface (MPI) communicationreal-time simulation

References

[1]

J. Jordan, T. Ippen, M. Helias, I. Kitayama, M. Sato, J. Igarashi, M. Diesmann, and S. Kunkel, Extremely scalable spiking neuronal network simulation code: From laptops to exascale computers, Front. Neuroinform., vol. 12, p. 2, 2018.
Crossref Google Scholar
[2]

S. Kunkel, M. Schmidt, J. M. Eppler, H. E. Plesser, G. Masumoto, J. Igarashi, S. Ishii, T. Fukai, A. Morrison, M. Diesmann et al., Spiking network simulation code for petascale computers, Front. Neuroinform., vol. 8, p. 78, 2014.
Crossref Google Scholar
[3]

S. Kunkel, T. C. Potjans, J. M. Eppler, H. E. Plesser, A. Morrison, and M. Diesmann, Meeting the memory challenges of brain-scale network simulation, Front. Neuroinform., vol. 5, p. 35, 2012.
Crossref Google Scholar
[4]

J. Igarashi, O. Shouno, T. Fukai, and H. Tsujino, Real-time simulation of a spiking neural network model of the basal Ganglia circuitry using general purpose computing on graphics processing units, Neural Netw., vol. 24, no. 9, pp. 950–960, 2011.
Crossref Google Scholar
[5]

M. Migliore, C. Cannia, W. W. Lytton, H. Markram, and M. L. Hines, Parallel network simulations with NEURON, J. Comput. Neurosci., vol. 21, no. 2, pp. 119–129, 2006.
Crossref Google Scholar
[6]
D. F. M. Goodman, The brian simulator, Front. Neurosci., vol. 3, no. 2, pp. 192–197, 2009.
Crossref
[7]
A. K. Fidjeland, E. B. Roesch, M. P. Shanahan, and W. Luk, NeMo: A platform for neural modelling of spiking neurons using GPUs, in Proc. 20th IEEE Int. Conf. Application-specific Systems, Architectures and Processors, Boston, MA, USA, 2009, pp. 137–144.
Crossref
[8]
M. Beyeler, K. D. Carlson, T.-S. Chou, N. Dutt, and J. L. Krichmar, CARLsim 3: A user-friendly and highly optimized library for the creation of neurobiologically detailed spiking neural networks, in Proc. Int. Joint Conf. Neural Networks (IJCNN), Killarney, Ireland, 2015, pp. 1–8.
Crossref
[9]
T. Nowotny, Flexible neuronal network simulation framework using code generation for NVidia® CUDA™, BMC Neurosci., vol. 12, no. 1, p. P239, 2011.
Crossref
[10]
K. Minkovich, C. M. Thibeault, M. J. O’Brien, A. Nogin, Y. Cho, and N. Srinivasa, HRLSim: A high performance spiking neural network simulator for GPGPU clusters, IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 2, pp. 316–331, 2014.
Crossref
[11]

M. Stimberg, D. F. M. Goodman, and T. Nowotny, Brian2GeNN: Accelerating spiking neural network simulations with graphics hardware, Sci. Rep., vol. 10, no. 1, p. 410, 2020.
Crossref Google Scholar
[12]
J. C. Knight and T. Nowotny, GPUs outperform current HPC and neuromorphic solutions in terms of speed and energy when simulating a highly-connected cortical model, Front. Neurosci., vol. 12, p. 941, 2018.
Crossref
[13]

E. Yavuz, J. Turner, and T. Nowotny, GeNN: A code generation framework for accelerated brain simulations, Sci. Rep., vol. 6, no. 1, p. 18854, 2016.
Crossref Google Scholar
[14]

R. Brette and D. F. M. Goodman, Simulating spiking neural networks on GPU, Netw. Comput. Neural Syst., vol. 23, no. 4, pp. 167–182, 2012.
Crossref Google Scholar
[15]

S. Henker, J. Partzsch, and R. Schüffny, Accuracy evaluation of numerical methods used in state-of-the-art simulators for spiking neural networks, J. Comput. Neurosci., vol. 32, no. 2, pp. 309–326, 2012.
Crossref Google Scholar
[16]

N. Imam and T. A. Cleland, Rapid online learning and robust recall in a neuromorphic olfactory circuit, Nat. Mach. Intell., vol. 2, no. 3, pp. 181–191, 2020.
Crossref Google Scholar
[17]

J. Pei, L. Deng, S. Song, M. Zhao, Y. Zhang, S. Wu, G. Wang, Z. Zou, Z. Wu, W. He et al., Towards artificial general intelligence with hybrid Tianjic chip architecture, Nature, vol. 572, no. 7767, pp. 106–111, 2019.
Crossref Google Scholar
[18]
N. Ahmad, J. B. Isbister, T. S. Smithe, and S. M. Stringer, Spike: AGPU optimised spiking neural network simulator, doi: https://doi.org/10.1101/461160.
Crossref
[19]

P. Qu, Y. Zhang, X. Fei, and W. Zheng, High performance simulation of spiking neural network on GPGPUs, IEEE Trans. Parallel Distrib. Syst., vol. 31, no. 11, pp. 2510–2523, 2020.
Crossref Google Scholar
[20]
D. Bautembach, I. Oikonomidis, and A. Argyros, Multi-GPU SNN simulation with static load balancing, arXiv preprint arXiv: 2102.04681, 2021.
Crossref
[21]
Z. Xu, J. Lin, and S. Matsuoka, Benchmarking SW26010 many-core processor, in Proc. IEEE Int. Parallel and Distributed Processing Symp. Workshops (IPDPSW), Lake Buena Vista, FL, USA, 2017, pp. 743–752.
Crossref
[22]

J. Lin, Z. Xu, L. Cai, A. Nukada, and S. Matsuoka, Evaluating the SW26010 many-core processor with a micro-benchmark suite for performance optimizations, Parallel Comput., vol. 77, pp. 128–143, 2018.
Crossref Google Scholar
[23]
L. F. Xu, C. D. Li, and L. Chen, Comparison and analysis for neuron models, (in Chinese), Acta Physica Sinica, vol. 65, no. 24, p. 240701, 2016.
Crossref
[24]
E. M. Izhikevich, Which model to use for cortical spiking neurons? IEEE Trans. Neural Netw., vol. 15, no. 5, pp. 1063–1070, 2004.
Crossref
[25]
E. M. Izhikevich, Simple model of spiking neurons, IEEE Trans. Neural Netw., vol. 14, no. 6, pp. 1569–1572, 2003.
Crossref
[26]

H. Markram, W. Gerstner, and P. J. Sjöström, Spike-timing-dependent plasticity: A comprehensive overview, Front. Synaptic Neurosci., vol. 4, p. 2, 2012.
Crossref Google Scholar
[27]

A. Morrison, A. Aertsen, and M. Diesmann, Spike-timing-dependent plasticity in balanced random networks, Neural Comput., vol. 19, no. 6, pp. 1437–1467, 2007.
Crossref Google Scholar
[28]

W. Gerstner, R. Kempter, J. L. van Hemmen, and H. Wagner, A neuronal learning rule for sub-millisecond temporal coding, Nature, vol. 383, no. 6595, pp. 76–81, 1996.
Crossref Google Scholar
[29]

H. Fu, J. Liao, J. Yang, L. Wang, Z. Song, X. Huang, C. Yang, W. Xue, F. Liu, F. Qiao, et al., The Sunway TaihuLight supercomputer: System and applications, Sci. China Inf. Sci., vol. 59, no. 7, p. 072001, 2016.
Crossref Google Scholar
[30]

X. Li, W. Wang, F. Xue, and Y. Song, Computational modeling of spiking neural network with learning rules from STDP and intrinsic plasticity, Phys. A Stat. Mech. Appl., vol. 491, pp. 716–728, 2018.
Crossref Google Scholar
[31]

H. Markram, W. Gerstner, and P. J. Sjöström, Spike-timing-dependent plasticity: A comprehensive overview, Front. Syn. Neurosci., vol. 4, p. 2, 2012.
Crossref Google Scholar
[32]
W. Dong, L. Kang, Z. Quan, K. Li, K. Li, Z. Hao, and X.-H. Xie, Implementing molecular dynamics simulation on sunway TaihuLight system, in Proc. IEEE 18th Int. Conf. High Performance Computing and Communications; IEEE 14th Int. Conf. Smart City; IEEE 2nd Int. Conf. Data Science and Systems (HPCC/SmartCity/DSS), Sydney, Australia, 2016, pp. 443-450.
Crossref
[33]
X. Duan, P. Gao, T. Zhang, M. Zhang, W. Liu, W. Zhang, W. Xue, H. Fu, L. Gan, D. Chen, et al., Redesigning LAMMPS for peta-scale and hundred-billion-atom simulation on sunway TaihuLight, in Proc. SC18 : Int. Conf. for High Performance Computing, Networking, Storage and Analysis, Dallas, TX, USA, 2018, pp. 148–159.
Crossref
[34]
J. Fang, H. Fu, W. Zhao, B. Chen, W. Zheng, and G. Yang, swDNN: A library for accelerating deep learning applications on sunway TaihuLight, in Proc. IEEE Int. Parallel and Distributed Processing Symp. (IPDPS), Orlando, FL, USA, 2017, pp. 615–624.
Crossref
[35]

M. J. Shelley and L. Tao, Efficient and accurate time-stepping schemes for integrate-and-fire neuronal networks, J. Comput. Neurosci., vol. 11, no. 2, pp. 111–119, 2001.
Crossref Google Scholar
[36]

R. D. Stewart and W. Bair, Spiking neural network simulation: Numerical integration with the Parker-Sochacki method, J. Comput. Neurosci., vol. 27, no. 1, pp. 115–133, 2009.
Crossref Google Scholar
[37]
S. Valadez-Godínez, H. Sossa, and R. Santiago-Montero, The step size impact on the computational cost of spiking neuron simulation, in Proc. Computing Conf., London, UK, 2017, pp. 722–728.
Crossref
[38]

E. M. Izhikevich and G. M. Edelman, Large-scale model of mammalian thalamocortical systems, Proc. Natl. Acad. Sci. U. S. A., vol. 105, no. 9, pp. 3593–3598, 2008.
Crossref Google Scholar
[39]

M. J. Skocik and L. N. Long, On the capabilities and computational costs of neuron models, IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 8, pp. 1474–1483, 2014.
Crossref Google Scholar
[40]
T. Zhang, L. Gan, H. Fu, W. Xue, and Z. Liu, SW_GROMACS: Accelerate GROMACS on Sunway TaihuLight, in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, doi:10.1145/3295500.3356190.
Crossref
[41]

T. C. Potjans and M. Diesmann, The cell-type specific cortical microcircuit: Relating structure and activity in a full-scale spiking network model, Cereb. Cortex, vol. 24, no. 3, pp. 785–806, 2014.
Crossref Google Scholar
Big Data Mining and Analytics
Volume 7 Issue 4,
December 2024
Pages 1301-1320
DOI: 10.26599/BDMA.2024.9020007
Cite this article:
Li X, Wang Z, Pan Y, et al. Distributed Heterogeneous Spiking Neural Network Simulator Using Sunway Accelerators. Big Data Mining and Analytics, 2024, 7(4): 1301-1320. https://doi.org/10.26599/BDMA.2024.9020007

173

Views

8

Downloads

0

Crossref

0

Web of Science

0

Scopus

0

CSCD

Altmetrics
Received: 12 October 2023
Revised: 19 January 2024
Accepted: 08 February 2024
Published: 04 December 2024
© The author(s) 2024.

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return