Show simple item record

dc.contributor.authorAlbasyoni, Alyazeed
dc.contributor.authorSafaryan, Mher
dc.contributor.authorCondat, Laurent
dc.contributor.authorRichtarik, Peter
dc.date.accessioned2020-10-13T11:59:04Z
dc.date.available2020-10-13T11:59:04Z
dc.date.issued2020-10-07
dc.identifier.urihttp://hdl.handle.net/10754/665557
dc.description.abstractCommunicating information, like gradient vectors, between computing nodes in distributed and federated learning is typically an unavoidable burden, resulting in scalability issues. Indeed, communication might be slow and costly. Recent advances in communication-efficient training algorithms have reduced this bottleneck by using compression techniques, in the form of sparsification, quantization, or low-rank approximation. Since compression is a lossy, or inexact, process, the iteration complexity is typically worsened; but the total communication complexity can improve significantly, possibly leading to large computation time savings. In this paper, we investigate the fundamental trade-off between the number of bits needed to encode compressed vectors and the compression error. We perform both worst-case and average-case analysis, providing tight lower bounds. In the worst-case analysis, we introduce an efficient compression operator, Sparse Dithering, which is very close to the lower bound. In the average-case analysis, we design a simple compression operator, Spherical Compression, which naturally achieves the lower bound. Thus, our new compression schemes significantly outperform the state of the art. We conduct numerical experiments to illustrate this improvement.
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/2010.03246
dc.rightsArchived with thanks to arXiv
dc.titleOptimal Gradient Compression for Distributed and Federated Learning
dc.typePreprint
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.eprint.versionPre-print
dc.identifier.arxivid2010.03246
kaust.personAlbasyoni, Alyazeed
kaust.personSafaryan, Mher
kaust.personCondat, Laurent
kaust.personRichtarik, Peter
refterms.dateFOA2020-10-13T11:59:30Z


Files in this item

Thumbnail
Name:
Preprintfile1.pdf
Size:
975.5Kb
Format:
PDF
Description:
Pre-print

This item appears in the following Collection(s)

Show simple item record