New Convergence Aspects of Stochastic Gradient Algorithms

Abstract

The classical convergence analysis of SGD is carried out under the assumptionthat the norm of the stochastic gradient is uniformly bounded. While this mighthold for some loss functions, it is violated for cases where the objectivefunction is strongly convex. In Bottou et al. (2016), a new analysis ofconvergence of SGD is performed under the assumption that stochastic gradientsare bounded with respect to the true gradient norm. We show that for stochasticproblems arising in machine learning such bound always holds; and we alsopropose an alternative convergence analysis of SGD with diminishing learningrate regime, which results in more relaxed conditions than those in Bottou etal. (2016). We then move on the asynchronous parallel setting, and proveconvergence of Hogwild! algorithm in the same regime in the case of diminishedlearning rate. It is well-known that SGD converges if a sequence of learningrates $\{\eta_t\}$ satisfies $\sum_{t=0}^\infty \eta_t \rightarrow \infty$ and$\sum_{t=0}^\infty \eta^2_t < \infty$. We show the convergence of SGD forstrongly convex objective function without using bounded gradient assumptionwhen $\{\eta_t\}$ is a diminishing sequence and $\sum_{t=0}^\infty \eta_t\rightarrow \infty$. In other words, we extend the current state-of-the-artclass of learning rates satisfying the convergence of SGD.