Error Compensated Loopless SVRG, Quartz, and SDCA for Distributed Optimization

dc.contributor.authorQian, Xun
dc.contributor.authorDong, Hanze
dc.contributor.authorRichtarik, Peter
dc.contributor.authorZhang, Tong
dc.contributor.departmentVisual Computing Center (VCC)
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Science and Engineering (CEMSE) Division
dc.contributor.institutionHKUST
dc.date.accessioned2021-10-04T13:07:47Z
dc.date.available2021-10-04T13:07:47Z
dc.date.issued2021-09-21
dc.description.abstractThe communication of gradients is a key bottleneck in distributed training of large scale machine learning models. In order to reduce the communication cost, gradient compression (e.g., sparsification and quantization) and error compensation techniques are often used. In this paper, we propose and study three new efficient methods in this space: error compensated loopless SVRG method (EC-LSVRG), error compensated Quartz (EC-Quartz), and error compensated SDCA (EC-SDCA). Our method is capable of working with any contraction compressor (e.g., TopK compressor), and we perform analysis for convex optimization problems in the composite case and smooth case for EC-LSVRG. We prove linear convergence rates for both cases and show that in the smooth case the rate has a better dependence on the parameter associated with the contraction compressor. Further, we show that in the smooth case, and under some certain conditions, error compensated loopless SVRG has the same convergence rate as the vanilla loopless SVRG method. Then we show that the convergence rates of EC-Quartz and EC-SDCA in the composite case are as good as EC-LSVRG in the smooth case. Finally, numerical experiments are presented to illustrate the efficiency of our methods.
dc.eprint.versionPre-print
dc.identifier.arxivid2109.10049
dc.identifier.urihttp://hdl.handle.net/10754/672105
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/2109.10049.pdf
dc.rightsArchived with thanks to arXiv
dc.titleError Compensated Loopless SVRG, Quartz, and SDCA for Distributed Optimization
dc.typePreprint
display.details.left<span><h5>Type</h5>Preprint<br><br><h5>Authors</h5><a href="https://repository.kaust.edu.sa/search?query=orcid.id:0000-0002-6072-2684&spc.sf=dc.date.issued&spc.sd=DESC">Qian, Xun</a> <a href="https://orcid.org/0000-0002-6072-2684" target="_blank"><img src="https://repository.kaust.edu.sa/server/api/core/bitstreams/82a625b4-ed4b-40c8-865a-d6a5225a26a4/content" width="16" height="16"/></a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Dong, Hanze,equals">Dong, Hanze</a><br><a href="https://repository.kaust.edu.sa/search?query=orcid.id:0000-0003-4380-5848&spc.sf=dc.date.issued&spc.sd=DESC">Richtarik, Peter</a> <a href="https://orcid.org/0000-0003-4380-5848" target="_blank"><img src="https://repository.kaust.edu.sa/server/api/core/bitstreams/82a625b4-ed4b-40c8-865a-d6a5225a26a4/content" width="16" height="16"/></a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.author=Zhang, Tong,equals">Zhang, Tong</a><br><br><h5>KAUST Department</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.department=Visual Computing Center (VCC),equals">Visual Computing Center (VCC)</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.department=Computer Science Program,equals">Computer Science Program</a><br><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.department=Computer, Electrical and Mathematical Science and Engineering (CEMSE) Division,equals">Computer, Electrical and Mathematical Science and Engineering (CEMSE) Division</a><br><br><h5>Date</h5>2021-09-21</span>
display.details.right<span><h5>Abstract</h5>The communication of gradients is a key bottleneck in distributed training of large scale machine learning models. In order to reduce the communication cost, gradient compression (e.g., sparsification and quantization) and error compensation techniques are often used. In this paper, we propose and study three new efficient methods in this space: error compensated loopless SVRG method (EC-LSVRG), error compensated Quartz (EC-Quartz), and error compensated SDCA (EC-SDCA). Our method is capable of working with any contraction compressor (e.g., TopK compressor), and we perform analysis for convex optimization problems in the composite case and smooth case for EC-LSVRG. We prove linear convergence rates for both cases and show that in the smooth case the rate has a better dependence on the parameter associated with the contraction compressor. Further, we show that in the smooth case, and under some certain conditions, error compensated loopless SVRG has the same convergence rate as the vanilla loopless SVRG method. Then we show that the convergence rates of EC-Quartz and EC-SDCA in the composite case are as good as EC-LSVRG in the smooth case. Finally, numerical experiments are presented to illustrate the efficiency of our methods.<br><br><h5>Publisher</h5><a href="https://repository.kaust.edu.sa/search?spc.sf=dc.date.issued&spc.sd=DESC&f.publisher=arXiv,equals">arXiv</a><br><br><h5>arXiv</h5><a href="https://arxiv.org/abs/2109.10049">2109.10049</a><br><br><h5>Additional Links</h5>https://arxiv.org/pdf/2109.10049.pdf</span>
kaust.personQian, Xun
kaust.personRichtarik, Peter
orcid.authorQian, Xun::0000-0002-6072-2684
orcid.authorDong, Hanze
orcid.authorRichtarik, Peter::0000-0003-4380-5848
orcid.authorZhang, Tong
orcid.id0000-0003-4380-5848
orcid.id0000-0002-6072-2684
refterms.dateFOA2021-10-04T13:08:54Z
Files
Original bundle
Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
2109.10049.pdf
Size:
2.37 MB
Format:
Adobe Portable Document Format
Description:
pre-print