Accelerated Stochastic Matrix Inversion: General Theory and Speeding up BFGS Rules for Faster Second-Order Optimization

Handle URI:
http://hdl.handle.net/10754/627161
Title:
Accelerated Stochastic Matrix Inversion: General Theory and Speeding up BFGS Rules for Faster Second-Order Optimization
Authors:
Gower, Robert M.; Hanzely, Filip; Richtarik, Peter; Stich, Sebastian
Abstract:
We present the first accelerated randomized algorithm for solving linear systems in Euclidean spaces. One essential problem of this type is the matrix inversion problem. In particular, our algorithm can be specialized to invert positive definite matrices in such a way that all iterates (approximate solutions) generated by the algorithm are positive definite matrices themselves. This opens the way for many applications in the field of optimization and machine learning. As an application of our general theory, we develop the {\em first accelerated (deterministic and stochastic) quasi-Newton updates}. Our updates lead to provably more aggressive approximations of the inverse Hessian, and lead to speed-ups over classical non-accelerated rules in numerical experiments. Experiments with empirical risk minimization show that our rules can accelerate training of machine learning models.
KAUST Department:
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division; Applied Mathematics and Computational Science Program; Computer Science Program
Publisher:
arXiv
Issue Date:
12-Feb-2018
ARXIV:
arXiv:1802.04079
Type:
Preprint
Additional Links:
http://arxiv.org/abs/1802.04079v1; http://arxiv.org/pdf/1802.04079v1
Appears in Collections:
Other/General Submission; Applied Mathematics and Computational Science Program; Computer Science Program; Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

Full metadata record

DC FieldValue Language
dc.contributor.authorGower, Robert M.en
dc.contributor.authorHanzely, Filipen
dc.contributor.authorRichtarik, Peteren
dc.contributor.authorStich, Sebastianen
dc.date.accessioned2018-02-22T10:34:40Z-
dc.date.available2018-02-22T10:34:40Z-
dc.date.issued2018-02-12en
dc.identifier.urihttp://hdl.handle.net/10754/627161-
dc.description.abstractWe present the first accelerated randomized algorithm for solving linear systems in Euclidean spaces. One essential problem of this type is the matrix inversion problem. In particular, our algorithm can be specialized to invert positive definite matrices in such a way that all iterates (approximate solutions) generated by the algorithm are positive definite matrices themselves. This opens the way for many applications in the field of optimization and machine learning. As an application of our general theory, we develop the {\em first accelerated (deterministic and stochastic) quasi-Newton updates}. Our updates lead to provably more aggressive approximations of the inverse Hessian, and lead to speed-ups over classical non-accelerated rules in numerical experiments. Experiments with empirical risk minimization show that our rules can accelerate training of machine learning models.en
dc.publisherarXiven
dc.relation.urlhttp://arxiv.org/abs/1802.04079v1en
dc.relation.urlhttp://arxiv.org/pdf/1802.04079v1en
dc.rightsArchived with thanks to arXiven
dc.titleAccelerated Stochastic Matrix Inversion: General Theory and Speeding up BFGS Rules for Faster Second-Order Optimizationen
dc.typePreprinten
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Divisionen
dc.contributor.departmentApplied Mathematics and Computational Science Programen
dc.contributor.departmentComputer Science Programen
dc.eprint.versionPre-printen
dc.contributor.institutionTelecom ParisTech, Paris, Franceen
dc.contributor.institutionMoscow Institute of Physics and Technology, Moscow, Russiaen
dc.contributor.institutionUniversity of Edinburgh, Edinburgh, United Kingdomen
dc.contributor.institutionEcole polytechnique federale de Lausanne (EPFL), Lausanne, Switzerlanden
dc.identifier.arxividarXiv:1802.04079en
kaust.authorHanzely, Filipen
kaust.authorRichtarik, Peteren
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.