Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems
KAUST DepartmentElectrical Engineering Program
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Physical Science and Engineering (PSE) Division
Office of the VP
Permanent link to this recordhttp://hdl.handle.net/10754/666254
MetadataShow full item record
AbstractTo tackle real-world challenges, deep and complex neural networks are generally used with a massive number of parameters, which require large memory size, extensive computational operations, and high energy consumption in neuromorphic hardware systems. In this work, we propose an unsupervised online adaptive weight pruning method that dynamically removes non-critical weights from a spiking neural network (SNN) to reduce network complexity and improve energy efficiency. The adaptive pruning method explores neural dynamics and firing activity of SNNs and adapts the pruning threshold over time and neurons during training. The proposed adaptation scheme allows the network to effectively identify critical weights associated with each neuron by changing the pruning threshold dynamically over time and neurons. It balances the connection strength of neurons with the previous layer with adaptive thresholds and prevents weak neurons from failure after pruning. We also evaluated improvement in the energy efficiency of SNNs with our method by computing synaptic operations (SOPs). Simulation results and detailed analyses have revealed that applying adaptation in the pruning threshold can significantly improve network performance and reduce the number of SOPs. The pruned SNN with 800 excitatory neurons can achieve a 30% reduction in SOPs during training and a 55% reduction during inference, with only 0.44% accuracy loss on MNIST dataset. Compared with a previously reported online soft pruning method, the proposed adaptive pruning method shows 3.33% higher classification accuracy and 67% more reduction in SOPs. The effectiveness of our method was confirmed on different datasets and for different network sizes. Our evaluation showed that the implementation overhead of the adaptive method regarding speed, area, and energy is negligible in the network. Therefore, this work offers a promising solution for effective network compression and building highly energy-efficient neuromorphic systems in real-time applications.
CitationGuo, W., Fouda, M. E., Yantir, H. E., Eltawil, A. M., & Salama, K. N. (2020). Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems. Frontiers in Neuroscience, 14. doi:10.3389/fnins.2020.598876
SponsorsWe acknowledge the financial support from King Abdullah University of Science and Technology (KAUST), Saudi Arabia. Funding. This research was funded by King Abdullah University of Science and Technology (KAUST) AI Initiative.
PublisherFrontiers Media SA
JournalFrontiers in Neuroscience
Except where otherwise noted, this item's license is described as This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.