Accelerated Gossip via Stochastic Heavy Ball Method

Abstract

In this paper we show how the stochastic heavy ball method (SHB) -- a popularmethod for solving stochastic convex and non-convex optimization problems--operates as a randomized gossip algorithm. In particular, we focus on twospecial cases of SHB: the Randomized Kaczmarz method with momentum and itsblock variant. Building upon a recent framework for the design and analysis ofrandomized gossip algorithms, [Loizou Richtarik, 2016] we interpret thedistributed nature of the proposed methods. We present novel protocols forsolving the average consensus problem where in each step all nodes of thenetwork update their values but only a subset of them exchange their privatevalues. Numerical experiments on popular wireless sensor networks showing thebenefits of our protocols are also presented.