SAGA with Arbitrary Sampling

Abstract

We study the problem of minimizing the average of a very large number ofsmooth functions, which is of key importance in training supervised learningmodels. One of the most celebrated methods in this context is the SAGAalgorithm. Despite years of research on the topic, a general-purpose version ofSAGA---one that would include arbitrary importance sampling and minibatchingschemes---does not exist. We remedy this situation and propose a general andflexible variant of SAGA following the {\em arbitrary sampling} paradigm. Weperform an iteration complexity analysis of the method, largely possible due tothe construction of new stochastic Lyapunov functions. We establish linearconvergence rates in the smooth and strongly convex regime, and under aquadratic functional growth condition (i.e., in a regime not assuming strongconvexity). Our rates match those of the primal-dual method Quartz for which anarbitrary sampling analysis is available, which makes a significant steptowards closing the gap in our understanding of complexity of primal and dualmethods for finite sum problems.