Stochastic Reformulations of Linear Systems: Algorithms and Convergence Theory

Handle URI:
http://hdl.handle.net/10754/626554
Title:
Stochastic Reformulations of Linear Systems: Algorithms and Convergence Theory
Authors:
Richtarik, Peter; Takáč, Martin
Abstract:
We develop a family of reformulations of an arbitrary consistent linear system into a stochastic problem. The reformulations are governed by two user-defined parameters: a positive definite matrix defining a norm, and an arbitrary discrete or continuous distribution over random matrices. Our reformulation has several equivalent interpretations, allowing for researchers from various communities to leverage their domain specific insights. In particular, our reformulation can be equivalently seen as a stochastic optimization problem, stochastic linear system, stochastic fixed point problem and a probabilistic intersection problem. We prove sufficient, and necessary and sufficient conditions for the reformulation to be exact. Further, we propose and analyze three stochastic algorithms for solving the reformulated problem---basic, parallel and accelerated methods---with global linear convergence rates. The rates can be interpreted as condition numbers of a matrix which depends on the system matrix and on the reformulation parameters. This gives rise to a new phenomenon which we call stochastic preconditioning, and which refers to the problem of finding parameters (matrix and distribution) leading to a sufficiently small condition number. Our basic method can be equivalently interpreted as stochastic gradient descent, stochastic Newton method, stochastic proximal point method, stochastic fixed point method, and stochastic projection method, with fixed stepsize (relaxation parameter), applied to the reformulations.
KAUST Department:
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division; Computer Science Program
Publisher:
arXiv
Issue Date:
4-Jun-2017
ARXIV:
arXiv:1706.01108
Type:
Preprint
Additional Links:
http://arxiv.org/abs/1706.01108v2; http://arxiv.org/pdf/1706.01108v2
Appears in Collections:
Other/General Submission; Computer Science Program; Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

Full metadata record

DC FieldValue Language
dc.contributor.authorRichtarik, Peteren
dc.contributor.authorTakáč, Martinen
dc.date.accessioned2017-12-28T07:32:16Z-
dc.date.available2017-12-28T07:32:16Z-
dc.date.issued2017-06-04en
dc.identifier.urihttp://hdl.handle.net/10754/626554-
dc.description.abstractWe develop a family of reformulations of an arbitrary consistent linear system into a stochastic problem. The reformulations are governed by two user-defined parameters: a positive definite matrix defining a norm, and an arbitrary discrete or continuous distribution over random matrices. Our reformulation has several equivalent interpretations, allowing for researchers from various communities to leverage their domain specific insights. In particular, our reformulation can be equivalently seen as a stochastic optimization problem, stochastic linear system, stochastic fixed point problem and a probabilistic intersection problem. We prove sufficient, and necessary and sufficient conditions for the reformulation to be exact. Further, we propose and analyze three stochastic algorithms for solving the reformulated problem---basic, parallel and accelerated methods---with global linear convergence rates. The rates can be interpreted as condition numbers of a matrix which depends on the system matrix and on the reformulation parameters. This gives rise to a new phenomenon which we call stochastic preconditioning, and which refers to the problem of finding parameters (matrix and distribution) leading to a sufficiently small condition number. Our basic method can be equivalently interpreted as stochastic gradient descent, stochastic Newton method, stochastic proximal point method, stochastic fixed point method, and stochastic projection method, with fixed stepsize (relaxation parameter), applied to the reformulations.en
dc.publisherarXiven
dc.relation.urlhttp://arxiv.org/abs/1706.01108v2en
dc.relation.urlhttp://arxiv.org/pdf/1706.01108v2en
dc.rightsArchived with thanks to arXiven
dc.titleStochastic Reformulations of Linear Systems: Algorithms and Convergence Theoryen
dc.typePreprinten
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Divisionen
dc.contributor.departmentComputer Science Programen
dc.eprint.versionPre-printen
dc.contributor.institutionUniversity of Edinburghen
dc.contributor.institutionLehigh Universityen
dc.identifier.arxividarXiv:1706.01108en
kaust.authorRichtarik, Peteren
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.