Randomized Block Cubic Newton Method

Handle URI:
http://hdl.handle.net/10754/627181
Title:
Randomized Block Cubic Newton Method
Authors:
Doikov, Nikita; Richtarik, Peter
Abstract:
We study the problem of minimizing the sum of three convex functions: a differentiable, twice-differentiable and a non-smooth term in a high dimensional setting. To this effect we propose and analyze a randomized block cubic Newton (RBCN) method, which in each iteration builds a model of the objective function formed as the sum of the natural models of its three components: a linear model with a quadratic regularizer for the differentiable term, a quadratic model with a cubic regularizer for the twice differentiable term, and perfect (proximal) model for the nonsmooth term. Our method in each iteration minimizes the model over a random subset of blocks of the search variable. RBCN is the first algorithm with these properties, generalizing several existing methods, matching the best known bounds in all special cases. We establish ${\cal O}(1/\epsilon)$, ${\cal O}(1/\sqrt{\epsilon})$ and ${\cal O}(\log (1/\epsilon))$ rates under different assumptions on the component functions. Lastly, we show numerically that our method outperforms the state-of-the-art on a variety of machine learning problems, including cubically regularized least-squares, logistic regression with constraints, and Poisson regression.
KAUST Department:
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division; Computer Science Program
Publisher:
arXiv
Issue Date:
12-Feb-2018
ARXIV:
arXiv:1802.04084
Type:
Preprint
Additional Links:
http://arxiv.org/abs/1802.04084v1; http://arxiv.org/pdf/1802.04084v1
Appears in Collections:
Other/General Submission; Computer Science Program; Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

Full metadata record

DC FieldValue Language
dc.contributor.authorDoikov, Nikitaen
dc.contributor.authorRichtarik, Peteren
dc.date.accessioned2018-02-22T10:34:42Z-
dc.date.available2018-02-22T10:34:42Z-
dc.date.issued2018-02-12en
dc.identifier.urihttp://hdl.handle.net/10754/627181-
dc.description.abstractWe study the problem of minimizing the sum of three convex functions: a differentiable, twice-differentiable and a non-smooth term in a high dimensional setting. To this effect we propose and analyze a randomized block cubic Newton (RBCN) method, which in each iteration builds a model of the objective function formed as the sum of the natural models of its three components: a linear model with a quadratic regularizer for the differentiable term, a quadratic model with a cubic regularizer for the twice differentiable term, and perfect (proximal) model for the nonsmooth term. Our method in each iteration minimizes the model over a random subset of blocks of the search variable. RBCN is the first algorithm with these properties, generalizing several existing methods, matching the best known bounds in all special cases. We establish ${\cal O}(1/\epsilon)$, ${\cal O}(1/\sqrt{\epsilon})$ and ${\cal O}(\log (1/\epsilon))$ rates under different assumptions on the component functions. Lastly, we show numerically that our method outperforms the state-of-the-art on a variety of machine learning problems, including cubically regularized least-squares, logistic regression with constraints, and Poisson regression.en
dc.publisherarXiven
dc.relation.urlhttp://arxiv.org/abs/1802.04084v1en
dc.relation.urlhttp://arxiv.org/pdf/1802.04084v1en
dc.rightsArchived with thanks to arXiven
dc.titleRandomized Block Cubic Newton Methoden
dc.typePreprinten
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Divisionen
dc.contributor.departmentComputer Science Programen
dc.eprint.versionPre-printen
dc.contributor.institutionNational Research University Higher School of Economics, Moscow, Russiaen
dc.contributor.institutionMoscow Institute of Physics and Technology, Dolgoprudny, Russia.en
dc.contributor.institutionUniversity of Edinburgh,Edinburgh, United Kingdomen
dc.identifier.arxividarXiv:1802.04084en
kaust.authorRichtarik, Peteren
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.