SGD and Hogwild! Convergence Without the Bounded Gradients Assumption

Abstract

Stochastic gradient descent (SGD) is the optimization algorithm of choice inmany machine learning applications such as regularized empirical riskminimization and training deep neural networks. The classical convergenceanalysis of SGD is carried out under the assumption that the norm of thestochastic gradient is uniformly bounded. While this might hold for some lossfunctions, it is always violated for cases where the objective function isstrongly convex. In (Bottou et al.,2016), a new analysis of convergence of SGDis performed under the assumption that stochastic gradients are bounded withrespect to the true gradient norm. Here we show that for stochastic problemsarising in machine learning such bound always holds; and we also propose analternative convergence analysis of SGD with diminishing learning rate regime,which results in more relaxed conditions than those in (Bottou et al.,2016). Wethen move on the asynchronous parallel setting, and prove convergence ofHogwild! algorithm in the same regime, obtaining the first convergence resultsfor this method in the case of diminished learning rate.