Nonconvex Variance Reduced Optimization with Arbitrary Sampling

Abstract

We provide the first importance sampling variants of variance reducedalgorithms for empirical risk minimization with non-convex loss functions. Inparticular, we analyze non-convex versions of SVRG, SAGA and SARAH. Our methodshave the capacity to speed up the training process by an order of magnitudecompared to the state of the art on real datasets. Moreover, we also improveupon current mini-batch analysis of these methods by proposing importancesampling for minibatches in this setting. Surprisingly, our approach can insome regimes lead to superlinear speedup with respect to the minibatch size,which is not usually present in stochastic optimization. All the above resultsfollow from a general analysis of the methods which works with arbitrarysampling, i.e., fully general randomized strategy for the selection of subsetsof examples to be sampled in each iteration. Finally, we also perform a novelimportance sampling analysis of SARAH in the convex setting.