On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning
dc.contributor.author | Dutta, Aritra | |
dc.contributor.author | Bergou, El Houcine | |
dc.contributor.author | Abdelmoniem, Ahmed M. | |
dc.contributor.author | Ho, Chen-Yu | |
dc.contributor.author | Sahu, Atal Narayan | |
dc.contributor.author | Canini, Marco | |
dc.contributor.author | Kalnis, Panos | |
dc.date.accessioned | 2020-07-28T06:15:03Z | |
dc.date.available | 2019-11-19T09:36:56Z | |
dc.date.available | 2020-07-28T06:15:03Z | |
dc.date.issued | 2020-04-03 | |
dc.identifier.citation | Dutta, A., Bergou, E. H., Abdelmoniem, A. M., Ho, C.-Y., Sahu, A. N., Canini, M., & Kalnis, P. (2020). On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 3817–3824. doi:10.1609/aaai.v34i04.5793 | |
dc.identifier.issn | 2374-3468 | |
dc.identifier.issn | 2159-5399 | |
dc.identifier.doi | 10.1609/aaai.v34i04.5793 | |
dc.identifier.uri | http://hdl.handle.net/10754/660127 | |
dc.description.abstract | Compressed communication, in the form of sparsification or quantization of stochastic gradients, is employed to reduce communication costs in distributed data-parallel training of deep neural networks. However, there exists a discrepancy between theory and practice: while theoretical analysis of most existing compression methods assumes compression is applied to the gradients of the entire model, many practical implementations operate individually on the gradients of each layer of the model.In this paper, we prove that layer-wise compression is, in theory, better, because the convergence rate is upper bounded by that of entire-model compression for a wide range of biased and unbiased compression methods. However, despite the theoretical bound, our experimental study of six well-known methods shows that convergence, in practice, may or may not be better, depending on the actual trained model and compression ratio. Our findings suggest that it would be advantageous for deep learning frameworks to include support for both layer-wise and entire-model compression. | |
dc.publisher | Association for the Advancement of Artificial Intelligence (AAAI) | |
dc.relation.url | https://aaai.org/ojs/index.php/AAAI/article/view/5793 | |
dc.rights | This is the technical report version of a paper later published in the Proceedings of the AAAI Conference on Artificial Intelligence | |
dc.subject | distributed deep learning | |
dc.subject | gradient compression | |
dc.subject | gradient sparsification | |
dc.subject | gradient quantization | |
dc.subject | layer-wise gradient compression | |
dc.title | On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning | |
dc.type | Conference Paper | |
dc.contributor.department | Computer Science | |
dc.contributor.department | Computer Science Program | |
dc.contributor.department | Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division | |
dc.contributor.department | InfoCloud Research Group | |
dc.identifier.journal | Proceedings of the AAAI Conference on Artificial Intelligence | |
dc.eprint.version | Pre-print | |
dc.contributor.institution | INRA | |
dc.identifier.volume | 34 | |
dc.identifier.issue | 04 | |
dc.identifier.pages | 3817-3824 | |
dc.identifier.arxivid | 1911.08250 | |
kaust.person | Dutta, Aritra | |
kaust.person | Abdelmoniem, Ahmed M. | |
kaust.person | Ho, Chen-Yu | |
kaust.person | Sahu, Atal Narayan | |
kaust.person | Canini, Marco | |
kaust.person | Kalnis, Panos | |
dc.relation.issupplementedby | github:sands-lab/layer-wise-aaai20 | |
refterms.dateFOA | 2019-11-19T00:00:00Z | |
display.relations | <b>Is Supplemented By:</b><br/> <ul><li><i>[Software]</i> <br/> Title: sands-lab/layer-wise-aaai20: Code repository for AAAI'20 paper: On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning. Publication Date: 2019-11-17. github: <a href="https://github.com/sands-lab/layer-wise-aaai20" >sands-lab/layer-wise-aaai20</a> Handle: <a href="http://hdl.handle.net/10754/667393" >10754/667393</a></a></li></ul> | |
dc.date.posted | 2019-11-19 |
Files in this item
This item appears in the following Collection(s)
-
Conference Papers
-
Computer Science Program
For more information visit: https://cemse.kaust.edu.sa/cs -
Computer, Electrical and Mathematical Science and Engineering (CEMSE) Division
For more information visit: https://cemse.kaust.edu.sa/