Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems
Type
ArticleKAUST Department
Electrical Engineering ProgramElectrical Engineering
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Physical Science and Engineering (PSE) Division
Office of the VP
Academic Affairs
Submitted Date
2020-08-25Permanent link to this record
http://hdl.handle.net/10754/666254
Metadata
Show full item recordAbstract
To tackle real-world challenges, deep and complex neural networks are generally used with a massive number of parameters, which require large memory size, extensive computational operations, and high energy consumption in neuromorphic hardware systems. In this work, we propose an unsupervised online adaptive weight pruning method that dynamically removes non-critical weights from a spiking neural network (SNN) to reduce network complexity and improve energy efficiency. The adaptive pruning method explores neural dynamics and firing activity of SNNs and adapts the pruning threshold over time and neurons during training. The proposed adaptation scheme allows the network to effectively identify critical weights associated with each neuron by changing the pruning threshold dynamically over time and neurons. It balances the connection strength of neurons with the previous layer with adaptive thresholds and prevents weak neurons from failure after pruning. We also evaluated improvement in the energy efficiency of SNNs with our method by computing synaptic operations (SOPs). Simulation results and detailed analyses have revealed that applying adaptation in the pruning threshold can significantly improve network performance and reduce the number of SOPs. The pruned SNN with 800 excitatory neurons can achieve a 30% reduction in SOPs during training and a 55% reduction during inference, with only 0.44% accuracy loss on MNIST dataset. Compared with a previously reported online soft pruning method, the proposed adaptive pruning method shows 3.33% higher classification accuracy and 67% more reduction in SOPs. The effectiveness of our method was confirmed on different datasets and for different network sizes. Our evaluation showed that the implementation overhead of the adaptive method regarding speed, area, and energy is negligible in the network. Therefore, this work offers a promising solution for effective network compression and building highly energy-efficient neuromorphic systems in real-time applications.Citation
ArraySponsors
We acknowledge the financial support from King Abdullah University of Science and Technology (KAUST), Saudi Arabia. Funding. This research was funded by King Abdullah University of Science and Technology (KAUST) AI Initiative.Publisher
Frontiers Media SAJournal
Frontiers in NeuroscienceAdditional Links
https://www.frontiersin.org/articles/10.3389/fnins.2020.598876/fullae974a485f413a2113503eed53cd6c53
10.3389/fnins.2020.598876
Scopus Count
Except where otherwise noted, this item's license is described as This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original
publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.