Stochastic Spectral and Conjugate Descent Methods

Handle URI:
http://hdl.handle.net/10754/627183
Title:
Stochastic Spectral and Conjugate Descent Methods
Authors:
Kovalev, Dmitry; Gorbunov, Eduard; Gasanov, Elnur; Richtarik, Peter
Abstract:
The state-of-the-art methods for solving optimization problems in big dimensions are variants of randomized coordinate descent (RCD). In this paper we introduce a fundamentally new type of acceleration strategy for RCD based on the augmentation of the set of coordinate directions by a few spectral or conjugate directions. As we increase the number of extra directions to be sampled from, the rate of the method improves, and interpolates between the linear rate of RCD and a linear rate independent of the condition number. We develop and analyze also inexact variants of these methods where the spectral and conjugate directions are allowed to be approximate only. We motivate the above development by proving several negative results which highlight the limitations of RCD with importance sampling.
KAUST Department:
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division; Computer Science Program
Publisher:
arXiv
Issue Date:
11-Feb-2018
ARXIV:
arXiv:1802.03703
Type:
Preprint
Additional Links:
http://arxiv.org/abs/1802.03703v1; http://arxiv.org/pdf/1802.03703v1
Appears in Collections:
Other/General Submission; Computer Science Program; Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

Full metadata record

DC FieldValue Language
dc.contributor.authorKovalev, Dmitryen
dc.contributor.authorGorbunov, Eduarden
dc.contributor.authorGasanov, Elnuren
dc.contributor.authorRichtarik, Peteren
dc.date.accessioned2018-02-22T10:34:43Z-
dc.date.available2018-02-22T10:34:43Z-
dc.date.issued2018-02-11en
dc.identifier.urihttp://hdl.handle.net/10754/627183-
dc.description.abstractThe state-of-the-art methods for solving optimization problems in big dimensions are variants of randomized coordinate descent (RCD). In this paper we introduce a fundamentally new type of acceleration strategy for RCD based on the augmentation of the set of coordinate directions by a few spectral or conjugate directions. As we increase the number of extra directions to be sampled from, the rate of the method improves, and interpolates between the linear rate of RCD and a linear rate independent of the condition number. We develop and analyze also inexact variants of these methods where the spectral and conjugate directions are allowed to be approximate only. We motivate the above development by proving several negative results which highlight the limitations of RCD with importance sampling.en
dc.publisherarXiven
dc.relation.urlhttp://arxiv.org/abs/1802.03703v1en
dc.relation.urlhttp://arxiv.org/pdf/1802.03703v1en
dc.rightsArchived with thanks to arXiven
dc.titleStochastic Spectral and Conjugate Descent Methodsen
dc.typePreprinten
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Divisionen
dc.contributor.departmentComputer Science Programen
dc.eprint.versionPre-printen
dc.contributor.institutionMoscow Institute of Physics and Technology, Dolgoprudny, Russiaen
dc.contributor.institutionUniversity of Edinburgh, Edinburgh, United Kingdom.en
dc.identifier.arxividarXiv:1802.03703en
kaust.authorRichtarik, Peteren
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.