99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it

Abstract

It is well known that many optimization methods, including SGD, SAGA, andAccelerated SGD for over-parameterized models, do not scale linearly in theparallel setting. In this paper, we present a new version of block coordinatedescent that solves this issue for a number of methods. The core idea is tomake the sampling of coordinate blocks on each parallel unit independent of theothers. Surprisingly, we prove that the optimal number of blocks to be updatedby each of $n$ units in every iteration is equal to $m/n$, where $m$ is thetotal number of blocks. As an illustration, this means that when $n=100$parallel units are used, $99\%$ of work is a waste of time. We demonstrate thatwith $m/n$ blocks used by each unit the iteration complexity often remains thesame. Among other applications which we mention, this fact can be exploited inthe setting of distributed optimization to break the communication bottleneck.Our claims are justified by numerical experiments which demonstrate almost aperfect match with our theory on a number of datasets.