KAUST DepartmentVisual Computing Center (VCC)
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Computer Science Program
Permanent link to this recordhttp://hdl.handle.net/10754/660285
MetadataShow full item record
AbstractWe develop and analyze a new family of nonaccelerated and accelerated loopless variancereduced methods for finite-sum optimization problems. Our convergence analysis relies on a novel expected smoothness condition which upper bounds the variance of the stochastic gradient estimation by a constant times a distance-like function. This allows us to handle with ease arbitrary sampling schemes as well as the nonconvex case. We perform an indepth estimation of these expected smoothness parameters and propose new importance samplings which allow linear speedup when the expected minibatch size is in a certain range. Furthermore, a connection between these expected smoothness parameters and expected separable overapproximation (ESO) is established, which allows us to exploit data sparsity as well. Our general methods and results recover as special cases the loopless SVRG (Hofmann et al., 2015) and loopless Katyusha (Kovalev et al., 2019) methods. Keywords: L-SVRG, L-Katyusha, Arbitrary sampling, Expected smoothness, ESO
SponsorsWe thank the action editor and two anonymous referees for their valuable comments. All authors are thankful for support through the KAUST Baseline Research Funding Scheme. Xun Qian and Peter Richtárik acknowledge further funding by the Extreme Computing Research Center at KAUST, and administrative support from the Visual Computing Center at KAUST. Zheng Qu acknowledges further funding by Hong Kong Research Grants Council Early Career Scheme 27303016.
Except where otherwise noted, this item's license is described as Archived with thanks to JOURNAL OF MACHINE LEARNING RESEARCH