Show simple item record

dc.contributor.authorGorbunov, Eduard
dc.contributor.authorHanzely, Filip
dc.contributor.authorRichtarik, Peter
dc.date.accessioned2020-11-11T11:43:02Z
dc.date.available2020-11-11T11:43:02Z
dc.date.issued2020-11-03
dc.identifier.urihttp://hdl.handle.net/10754/665897
dc.description.abstractWe present a unified framework for analyzing local SGD methods in the convex and strongly convex regimes for distributed/federated training of supervised machine learning models. We recover several known methods as a special case of our general framework, including Local-SGD/FedAvg, SCAFFOLD, and several variants of SGD not originally designed for federated learning. Our framework covers both the identical and heterogeneous data settings, supports both random and deterministic number of local steps, and can work with a wide array of local stochastic gradient estimators, including shifted estimators which are able to adjust the fixed points of local iterations for faster convergence. As an application of our framework, we develop multiple novel FL optimizers which are superior to existing methods. In particular, we develop the first linearly converging local SGD method which does not require any data homogeneity or other strong assumptions.
dc.description.sponsorshipThis work was supported by the KAUST baseline research grant of P. Richt´arik. Part of this work was done while E. Gorbunov was a research intern at KAUST. The research of E. Gorbunov was also partially supported by the Ministry of Science and Higher Education of the Russian Federation (Goszadaniye) 075-00337-20-03 and RFBR, project number 19-31-51001.
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/2011.02828
dc.rightsArchived with thanks to arXiv
dc.titleLocal SGD: Unified Theory and New Efficient Methods
dc.typePreprint
dc.contributor.departmentApplied Mathematics and Computational Science Program
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.eprint.versionPre-print
dc.contributor.institutionMIPT, Yandex, Sirius, Russia.
dc.identifier.arxivid2011.02828
kaust.personGorbunov, Eduard
kaust.personHanzely, Filip
kaust.personRichtarik, Peter
refterms.dateFOA2020-11-11T11:44:39Z
kaust.acknowledged.supportUnitKAUST baseline research


Files in this item

Thumbnail
Name:
Preprintfile1.pdf
Size:
1.948Mb
Format:
PDF
Description:
Pre-print

This item appears in the following Collection(s)

Show simple item record