Show simple item record

dc.contributor.authorKostopoulou, Kelly
dc.contributor.authorXu, Hang
dc.contributor.authorDutta, Aritra
dc.contributor.authorLi, Xin
dc.contributor.authorNtoulas, Alexandros
dc.contributor.authorKalnis, Panos
dc.date.accessioned2021-02-17T10:54:05Z
dc.date.available2021-02-17T10:54:05Z
dc.date.issued2021-02-05
dc.identifier.urihttp://hdl.handle.net/10754/667496
dc.description.abstractSparse tensors appear frequently in distributed deep learning, either as a direct artifact of the deep neural network's gradients, or as a result of an explicit sparsification process. Existing communication primitives are agnostic to the peculiarities of deep learning; consequently, they impose unnecessary communication overhead. This paper introduces DeepReduce, a versatile framework for the compressed communication of sparse tensors, tailored for distributed deep learning. DeepReduce decomposes sparse tensors in two sets, values and indices, and allows both independent and combined compression of these sets. We support a variety of common compressors, such as Deflate for values, or run-length encoding for indices. We also propose two novel compression schemes that achieve superior results: curve fitting-based for values and bloom filter-based for indices. DeepReduce is orthogonal to existing gradient sparsifiers and can be applied in conjunction with them, transparently to the end-user, to significantly lower the communication overhead. As proof of concept, we implement our approach on Tensorflow and PyTorch. Our experiments with large real models demonstrate that DeepReduce transmits fewer data and imposes lower computational overhead than existing methods, without affecting the training accuracy.
dc.description.sponsorshipKelly Kostopoulou was supported by the KAUST Visiting Student Research Program. The computing infrastructure was provided by the KAUST Super-computing Lab (KSL).
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/2102.03112
dc.rightsArchived with thanks to arXiv
dc.titleDeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning
dc.typePreprint
dc.contributor.departmentComputer Science
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.contributor.departmentInfoCloud Research Group
dc.contributor.departmentMaterial Science and Engineering Program
dc.contributor.departmentPhysical Science and Engineering (PSE) Division
dc.eprint.versionPre-print
dc.contributor.institutionColumbia University New York, NY, USA.
dc.contributor.institutionUniversity of Central Florida, Orlando. FL, USA.
dc.contributor.institutionNational and Kapodistrian, University of Athens, Greece.
dc.identifier.arxivid2102.03112
kaust.personXu, Hang
kaust.personDutta, Aritra
kaust.personKalnis, Panos
refterms.dateFOA2021-02-17T10:55:06Z


Files in this item

Thumbnail
Name:
Preprintfile1.pdf
Size:
3.313Mb
Format:
PDF
Description:
Pre-print

This item appears in the following Collection(s)

Show simple item record