Stochastic quasi-gradient methods: variance reduction via Jacobian sketching
dc.contributor.author | Gower, Robert M. | |
dc.contributor.author | Richtarik, Peter | |
dc.contributor.author | Bach, Francis | |
dc.date.accessioned | 2019-04-28T13:13:07Z | |
dc.date.available | 2019-04-28T13:13:07Z | |
dc.date.issued | 2020-05-12 | |
dc.date.submitted | 2018-05-08 | |
dc.identifier.citation | Gower, R. M., Richtárik, P., & Bach, F. (2020). Stochastic quasi-gradient methods: variance reduction via Jacobian sketching. Mathematical Programming. doi:10.1007/s10107-020-01506-0 | |
dc.identifier.issn | 1436-4646 | |
dc.identifier.issn | 0025-5610 | |
dc.identifier.doi | 10.1007/s10107-020-01506-0 | |
dc.identifier.uri | http://hdl.handle.net/10754/632522 | |
dc.description.abstract | We develop a new family of variance reduced stochastic gradient descent methods for minimizing the average of a very large number of smooth functions. Our method—JacSketch—is motivated by novel developments in randomized numerical linear algebra, and operates by maintaining a stochastic estimate of a Jacobian matrix composed of the gradients of individual functions. In each iteration, JacSketch efficiently updates the Jacobian matrix by first obtaining a random linear measurement of the true Jacobian through (cheap) sketching, and then projecting the previous estimate onto the solution space of a linear matrix equation whose solutions are consistent with the measurement. The Jacobian estimate is then used to compute a variance-reduced unbiased estimator of the gradient. Our strategy is analogous to the way quasi-Newton methods maintain an estimate of the Hessian, and hence our method can be seen as a stochastic quasi-gradient method. Our method can also be seen as stochastic gradient descent applied to a controlled stochastic optimization reformulation of the original problem, where the control comes from the Jacobian estimates. We prove that for smooth and strongly convex functions, JacSketch converges linearly with a meaningful rate dictated by a single convergence theorem which applies to general sketches. We also provide a refined convergence theorem which applies to a smaller class of sketches, featuring a novel proof technique based on a stochastic Lyapunov function. This enables us to obtain sharper complexity results for variants of JacSketch with importance sampling. By specializing our general approach to specific sketching strategies, JacSketch reduces to the celebrated stochastic average gradient (SAGA) method, and its several existing and many new minibatch, reduced memory, and importance sampling variants. Our rate for SAGA with importance sampling is the current best-known rate for this method, resolving a conjecture by Schmidt et al. (Proceedings of the eighteenth international conference on artificial intelligence and statistics, AISTATS 2015, San Diego, California, 2015). The rates we obtain for minibatch SAGA are also superior to existing rates and are sufficiently tight as to show a decrease in total complexity as the minibatch size increases. Moreover, we obtain the first minibatch SAGA method with importance sampling. | |
dc.description.sponsorship | Funding was provided by Fondation de Sciences Mathématiques de Paris, European Research Council (Grant No. ERC SEQUOIA), LabEx LMH (Grant No. ANR-11-LABX-0056-LMH). | |
dc.publisher | Springer Nature | |
dc.relation.url | http://link.springer.com/10.1007/s10107-020-01506-0 | |
dc.relation.url | https://link.springer.com/content/pdf/10.1007/s10107-020-01506-0.pdf | |
dc.rights | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. | |
dc.rights.uri | https://creativecommons.org/licenses/by/4.0 | |
dc.title | Stochastic quasi-gradient methods: variance reduction via Jacobian sketching | |
dc.type | Article | |
dc.contributor.department | Computer Science Program | |
dc.contributor.department | Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division | |
dc.identifier.journal | Mathematical Programming | |
dc.eprint.version | Publisher's Version/PDF | |
dc.contributor.institution | LTCI, Telécom Paris, Institut Polytechnique de Paris, Palaiseau, France | |
dc.contributor.institution | University of Edinburgh, Edinburgh, UK | |
dc.contributor.institution | Moscow Institute of Physics and Technology (MIPT), Dolgoprudny, Russia | |
dc.contributor.institution | INRIA - ENS - PSL Research University, Paris, France | |
dc.identifier.arxivid | 1805.02632 | |
kaust.person | Richtarik, Peter | |
dc.date.accepted | 2020-04-09 | |
dc.version | v1 | |
dc.identifier.eid | 2-s2.0-85084639279 | |
refterms.dateFOA | 2019-04-29T06:46:37Z | |
dc.date.posted | 2018-05-07 |
Files in this item
This item appears in the following Collection(s)
-
Articles
-
Computer Science Program
For more information visit: https://cemse.kaust.edu.sa/cs -
Computer, Electrical and Mathematical Science and Engineering (CEMSE) Division
For more information visit: https://cemse.kaust.edu.sa/