Show simple item record

dc.contributor.authorHorvath, Samuel
dc.contributor.authorLaskaridis, Stefanos
dc.contributor.authorAlmeida, Mario
dc.contributor.authorLeontiadis, Ilias
dc.contributor.authorVenieris, Stylianos I.
dc.contributor.authorLane, Nicholas D.
dc.date.accessioned2021-03-10T06:37:36Z
dc.date.available2021-03-10T06:37:36Z
dc.date.issued2021-02-26
dc.identifier.urihttp://hdl.handle.net/10754/668024
dc.description.abstractFederated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact, and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform limit on the model's capacity, restricted by the least capable participants. In this work, we introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in Neural Networks and enables the extraction of lower footprint submodels without the need of retraining. We further show that for linear maps our Ordered Dropout is equivalent to SVD. We employ this technique, along with a self-distillation methodology, in the realm of FL in a framework called FjORD. FjORD alleviates the problem of client system heterogeneity by tailoring the model width to the client's capabilities. Extensive evaluation on both CNNs and RNNs across diverse modalities shows that FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/2102.13451.pdf
dc.rightsArchived with thanks to arXiv
dc.titleFjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
dc.typePreprint
dc.contributor.departmentStatistics Program
dc.contributor.departmentStatistics
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.eprint.versionPre-print
dc.contributor.institutionSamsung AI Center, Cambridge, UK
dc.contributor.institutionUniversity of Cambridge, UK.
dc.identifier.arxivid2102.13451
kaust.personHorvath, Samuel
refterms.dateFOA2021-03-10T06:38:40Z


Files in this item

Thumbnail
Name:
Preprintfile1.pdf
Size:
10.15Mb
Format:
PDF
Description:
Pre-print

This item appears in the following Collection(s)

Show simple item record