Supervised Local Training with Backward Links for Deep Neural Networks

Abstract
The restricted training pattern in the standard BP requires end-to-end error propagation, causing large memory costs and prohibiting model parallelization. Existing local training methods aim to resolve the training obstacles by completely cutting off the backward path between modules and isolating their gradients. These methods prevent information exchange between modules and result in inferior performance. This work proposes a novel local training algorithm, BackLink, which introduces inter-module backward dependency and facilitates information to flow backward along with the network. To preserve the computational advantage of local training, BackLink restricts the error propagation length within the module. Extensive experiments performed in various deep convolutional neural networks demonstrate that our method consistently improves the classification performance of local training algorithms over other methods. For example, our method can surpass the conventional greedy local training method by 6.45% in accuracy in ResNet32 classifying CIFAR100 and recent work by 2.58% in ResNet110 classifying STL-10 with much lower complexity, respectively. Analysis of computational costs reveals that small overheads are incurred in GPU memory costs and runtime on multiple GPUs. Our method can lead up to a 79% reduction in memory cost and 52% in simulation runtime in ResNet110 compared to the standard BP. Therefore, our method could create new opportunities for improving training algorithms towards better efficiency for real-time learning applications.

Citation
Guo, W., Fouda, M. E., Eltawil, A. M., & Salama, K. N. (2023). Supervised Local Training with Backward Links for Deep Neural Networks. IEEE Transactions on Artificial Intelligence, 1–14. https://doi.org/10.1109/tai.2023.3251313

Acknowledgements
This work was funded by the King Abdullah University of Science and Technology (KAUST) AI Initiative, Saudi Arabia.

Publisher
Institute of Electrical and Electronics Engineers (IEEE)

Journal
IEEE Transactions on Artificial Intelligence

DOI
10.1109/tai.2023.3251313

Additional Links
https://ieeexplore.ieee.org/document/10058021/

Permanent link to this record