Show simple item record

dc.contributor.authorGower, Robert M.
dc.contributor.authorRichtarik, Peter
dc.contributor.authorBach, Francis
dc.date.accessioned2019-04-28T13:13:07Z
dc.date.available2019-04-28T13:13:07Z
dc.date.issued2020-05-12
dc.date.submitted2018-05-08
dc.identifier.citationGower, R. M., Richtárik, P., & Bach, F. (2020). Stochastic quasi-gradient methods: variance reduction via Jacobian sketching. Mathematical Programming. doi:10.1007/s10107-020-01506-0
dc.identifier.issn1436-4646
dc.identifier.issn0025-5610
dc.identifier.doi10.1007/s10107-020-01506-0
dc.identifier.urihttp://hdl.handle.net/10754/632522
dc.description.abstractWe develop a new family of variance reduced stochastic gradient descent methods for minimizing the average of a very large number of smooth functions. Our method—JacSketch—is motivated by novel developments in randomized numerical linear algebra, and operates by maintaining a stochastic estimate of a Jacobian matrix composed of the gradients of individual functions. In each iteration, JacSketch efficiently updates the Jacobian matrix by first obtaining a random linear measurement of the true Jacobian through (cheap) sketching, and then projecting the previous estimate onto the solution space of a linear matrix equation whose solutions are consistent with the measurement. The Jacobian estimate is then used to compute a variance-reduced unbiased estimator of the gradient. Our strategy is analogous to the way quasi-Newton methods maintain an estimate of the Hessian, and hence our method can be seen as a stochastic quasi-gradient method. Our method can also be seen as stochastic gradient descent applied to a controlled stochastic optimization reformulation of the original problem, where the control comes from the Jacobian estimates. We prove that for smooth and strongly convex functions, JacSketch converges linearly with a meaningful rate dictated by a single convergence theorem which applies to general sketches. We also provide a refined convergence theorem which applies to a smaller class of sketches, featuring a novel proof technique based on a stochastic Lyapunov function. This enables us to obtain sharper complexity results for variants of JacSketch with importance sampling. By specializing our general approach to specific sketching strategies, JacSketch reduces to the celebrated stochastic average gradient (SAGA) method, and its several existing and many new minibatch, reduced memory, and importance sampling variants. Our rate for SAGA with importance sampling is the current best-known rate for this method, resolving a conjecture by Schmidt et al. (Proceedings of the eighteenth international conference on artificial intelligence and statistics, AISTATS 2015, San Diego, California, 2015). The rates we obtain for minibatch SAGA are also superior to existing rates and are sufficiently tight as to show a decrease in total complexity as the minibatch size increases. Moreover, we obtain the first minibatch SAGA method with importance sampling.
dc.description.sponsorshipFunding was provided by Fondation de Sciences Mathématiques de Paris, European Research Council (Grant No. ERC SEQUOIA), LabEx LMH (Grant No. ANR-11-LABX-0056-LMH).
dc.publisherSpringer Nature
dc.relation.urlhttp://link.springer.com/10.1007/s10107-020-01506-0
dc.relation.urlhttps://link.springer.com/content/pdf/10.1007/s10107-020-01506-0.pdf
dc.rightsThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
dc.rights.urihttps://creativecommons.org/licenses/by/4.0
dc.titleStochastic quasi-gradient methods: variance reduction via Jacobian sketching
dc.typeArticle
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.identifier.journalMathematical Programming
dc.eprint.versionPublisher's Version/PDF
dc.contributor.institutionLTCI, Telécom Paris, Institut Polytechnique de Paris, Palaiseau, France
dc.contributor.institutionUniversity of Edinburgh, Edinburgh, UK
dc.contributor.institutionMoscow Institute of Physics and Technology (MIPT), Dolgoprudny, Russia
dc.contributor.institutionINRIA - ENS - PSL Research University, Paris, France
dc.identifier.arxivid1805.02632
kaust.personRichtarik, Peter
dc.date.accepted2020-04-09
dc.versionv1
dc.identifier.eid2-s2.0-85084639279
refterms.dateFOA2019-04-29T06:46:37Z
dc.date.posted2018-05-07


Files in this item

Thumbnail
Name:
Gower2020_Article_StochasticQuasi-gradientMethod.pdf
Size:
1.136Mb
Format:
PDF
Description:
Published version

This item appears in the following Collection(s)

Show simple item record

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Except where otherwise noted, this item's license is described as This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.