Show simple item record

dc.contributor.authorMalinovsky, Grigory
dc.contributor.authorRichtarik, Peter
dc.date.accessioned2022-05-12T07:48:54Z
dc.date.available2022-05-12T07:48:54Z
dc.date.issued2022-05-10
dc.identifier.urihttp://hdl.handle.net/10754/676866
dc.description.abstractRandom Reshuffling (RR), which is a variant of Stochastic Gradient Descent (SGD) employing sampling without replacement, is an immensely popular method for training supervised machine learning models via empirical risk minimization. Due to its superior practical performance, it is embedded and often set as default in standard machine learning software. Under the name FedRR, this method was recently shown to be applicable to federated learning (Mishchenko et al.,2021), with superior performance when compared to common baselines such as Local SGD. Inspired by this development, we design three new algorithms to improve FedRR further: compressed FedRR and two variance reduced extensions: one for taming the variance coming from shuffling and the other for taming the variance due to compression. The variance reduction mechanism for compression allows us to eliminate dependence on the compression parameter, and applying additional controlled linear perturbations for Random Reshuffling, introduced by Malinovsky et al.(2021) helps to eliminate variance at the optimum. We provide the first analysis of compressed local methods under standard assumptions without bounded gradient assumptions and for heterogeneous data, overcoming the limitations of the compression operator. We corroborate our theoretical results with experiments on synthetic and real data sets.
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/2205.03914.pdf
dc.rightsArchived with thanks to arXiv
dc.titleFederated Random Reshuffling with Compression and Variance Reduction
dc.typePreprint
dc.contributor.departmentKing Abdullah University of Science and Technology, KAUST
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Science and Engineering (CEMSE) Division
dc.eprint.versionPre-print
dc.identifier.arxivid2205.03914
kaust.personMalinovsky, Grigory
kaust.personRichtarik, Peter
refterms.dateFOA2022-05-12T07:49:48Z


Files in this item

Thumbnail
Name:
2205.03914.pdf
Size:
864.7Kb
Format:
PDF
Description:
Preprint

This item appears in the following Collection(s)

Show simple item record