Show simple item record

dc.contributor.authorMishchenko, Konstantin
dc.contributor.authorRichtarik, Peter
dc.date.accessioned2019-05-29T07:54:21Z
dc.date.available2019-05-29T07:54:21Z
dc.date.issued2018-10-31
dc.identifier.urihttp://hdl.handle.net/10754/653118
dc.description.abstractThe last decade witnessed a rise in the importance of supervised learningapplications involving {\em big data} and {\em big models}. Big data refers tosituations where the amounts of training data available and needed causesdifficulties in the training phase of the pipeline. Big model refers tosituations where large dimensional and over-parameterized models are needed forthe application at hand. Both of these phenomena lead to a dramatic increase inresearch activity aimed at taming the issues via the design of newsophisticated optimization algorithms. In this paper we turn attention to the{\em big constraints} scenario and argue that elaborate machine learningsystems of the future will necessarily need to account for a large number ofreal-world constraints, which will need to be incorporated in the trainingprocess. This line of work is largely unexplored, and provides ampleopportunities for future work and applications. To handle the {\em bigconstraints} regime, we propose a {\em stochastic penalty} formulation which{\em reduces the problem to the well understood big data regime}. Ourformulation has many interesting properties which relate it to the originalproblem in various ways, with mathematical guarantees. We give a number ofresults specialized to nonconvex loss functions, smooth convex functions,strongly convex functions and convex constraints. We show through experimentsthat our approach can beat competing approaches by several orders of magnitudewhen a medium accuracy solution is required.
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/1810.13387
dc.rightsArchived with thanks to arXiv
dc.titleA Stochastic Penalty Model for Convex and Nonconvex Optimization with Big Constraints
dc.typePreprint
dc.contributor.departmentComputer Science
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.eprint.versionPre-print
dc.contributor.institutionUniversity of Edinburgh
dc.contributor.institutionMoscow Institute of Physics and Technology
dc.identifier.arxivid1810.13387
kaust.personMishchenko, Konstantin
kaust.personRichtarik, Peter
dc.versionv1
refterms.dateFOA2019-05-29T07:54:53Z


Files in this item

Thumbnail
Name:
1810.13387.pdf
Size:
5.885Mb
Format:
PDF
Description:
Preprint

This item appears in the following Collection(s)

Show simple item record