Show simple item record

dc.contributor.authorHorvath, Samuel
dc.contributor.authorRichtarik, Peter
dc.date.accessioned2020-06-29T07:50:16Z
dc.date.available2020-06-29T07:50:16Z
dc.date.issued2020-06-19
dc.identifier.urihttp://hdl.handle.net/10754/663911
dc.description.abstractModern large-scale machine learning applications require stochastic optimization algorithms to be implemented on distributed compute systems. A key bottleneck of such systems is the communication overhead for exchanging information across the workers, such as stochastic gradients. Among the many techniques proposed to remedy this issue, one of the most successful is the framework of compressed communication with error feedback (EF). EF remains the only known technique that can deal with the error induced by contractive compressors which are not unbiased, such as Top-$K$. In this paper, we propose a new and theoretically and practically better alternative to EF for dealing with contractive compressors. In particular, we propose a construction which can transform any contractive compressor into an induced unbiased compressor. Following this transformation, existing methods able to work with unbiased compressors can be applied. We show that our approach leads to vast improvements over EF, including reduced memory requirements, better communication complexity guarantees and fewer assumptions. We further extend our results to federated learning with partial participation following an arbitrary distribution over the nodes, and demonstrate the benefits thereof. We perform several numerical experiments which validate our theoretical findings.
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/2006.11077
dc.rightsArchived with thanks to arXiv
dc.titleA Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
dc.typePreprint
dc.contributor.departmentStatistics Program
dc.contributor.departmentStatistics
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.eprint.versionPre-print
dc.identifier.arxivid2006.11077
kaust.personHorvath, Samuel
kaust.personRichtarik, Peter
refterms.dateFOA2020-06-29T07:51:05Z


Files in this item

Thumbnail
Name:
Preprintfile1.pdf
Size:
4.168Mb
Format:
PDF
Description:
Pre-print

This item appears in the following Collection(s)

Show simple item record