Show simple item record

dc.contributor.authorChen, Wenlin
dc.contributor.authorHorvath, Samuel
dc.contributor.authorRichtarik, Peter
dc.date.accessioned2020-11-03T13:49:58Z
dc.date.available2020-11-03T13:49:58Z
dc.date.issued2020-10-26
dc.identifier.urihttp://hdl.handle.net/10754/665795
dc.description.abstractIt is well understood that client-master communication can be a primary bottleneck in Federated Learning. In this work, we address this issue with a novel client subsampling scheme, where we restrict the number of clients allowed to communicate their updates back to the master node. In each communication round, all participated clients compute their updates, but only the ones with "important" updates communicate back to the master. We show that importance can be measured using only the norm of the update and we give a formula for optimal client participation. This formula minimizes the distance between the full update, where all clients participate, and our limited update, where the number of participating clients is restricted. In addition, we provide a simple algorithm that approximates the optimal formula for client participation which only requires secure aggregation and thus does not compromise client privacy. We show both theoretically and empirically that our approach leads to superior performance for Distributed SGD (DSGD) and Federated Averaging (FedAvg) compared to the baseline where participating clients are sampled uniformly. Finally, our approach is orthogonal to and compatible with existing methods for reducing communication overhead, such as local methods and communication compression methods.
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/2010.13723
dc.rightsArchived with thanks to arXiv
dc.titleOptimal Client Sampling for Federated Learning
dc.typePreprint
dc.contributor.departmentStatistics Program
dc.contributor.departmentStatistics
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.contributor.departmentComputer Science Program
dc.eprint.versionPre-print
dc.contributor.institutionUniversity of Cambridge Cambridge, United Kingdom.
dc.identifier.arxivid2010.13723
kaust.personHorvath, Samuel
kaust.personRichtarik, Peter
refterms.dateFOA2020-11-03T13:50:27Z


Files in this item

Thumbnail
Name:
Preprintfile1.pdf
Size:
811.4Kb
Format:
PDF
Description:
Pre-print

This item appears in the following Collection(s)

Show simple item record