SGD: General Analysis and Improved Rates

Abstract

We propose a general yet simple theorem describing the convergence of SGDunder the arbitrary sampling paradigm. Our theorem describes the convergence ofan infinite array of variants of SGD, each of which is associated with aspecific probability law governing the data selection rule used to formmini-batches. This is the first time such an analysis is performed, and most ofour variants of SGD were never explicitly considered in the literature before.Our analysis relies on the recently introduced notion of expected smoothnessand does not rely on a uniform bound on the variance of the stochasticgradients. By specializing our theorem to different mini-batching strategies,such as sampling with replacement and independent sampling, we derive exactexpressions for the stepsize as a function of the mini-batch size. With this wecan also determine the mini-batch size that optimizes the total complexity, andshow explicitly that as the variance of the stochastic gradient evaluated atthe minimum grows, so does the optimal mini-batch size. For zero variance, theoptimal mini-batch size is one. Moreover, we prove insightfulstepsize-switching rules which describe when one should switch from a constantto a decreasing stepsize regime.