Show simple item record

dc.contributor.authorHorvath, Samuel
dc.contributor.authorRichtarik, Peter
dc.date.accessioned2019-05-29T07:42:56Z
dc.date.available2019-05-29T07:42:56Z
dc.date.issued2018-09-11
dc.identifier.urihttp://hdl.handle.net/10754/653115
dc.description.abstractWe provide the first importance sampling variants of variance reducedalgorithms for empirical risk minimization with non-convex loss functions. Inparticular, we analyze non-convex versions of SVRG, SAGA and SARAH. Our methodshave the capacity to speed up the training process by an order of magnitudecompared to the state of the art on real datasets. Moreover, we also improveupon current mini-batch analysis of these methods by proposing importancesampling for minibatches in this setting. Surprisingly, our approach can insome regimes lead to superlinear speedup with respect to the minibatch size,which is not usually present in stochastic optimization. All the above resultsfollow from a general analysis of the methods which works with arbitrarysampling, i.e., fully general randomized strategy for the selection of subsetsof examples to be sampled in each iteration. Finally, we also perform a novelimportance sampling analysis of SARAH in the convex setting.
dc.publisherarXiv
dc.relation.urlhttps://arxiv.org/pdf/1809.04146
dc.rightsArchived with thanks to arXiv
dc.titleNonconvex Variance Reduced Optimization with Arbitrary Sampling
dc.typePreprint
dc.contributor.departmentComputer Science
dc.contributor.departmentComputer Science Program
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
dc.contributor.departmentStatistics
dc.contributor.departmentStatistics Program
dc.eprint.versionPre-print
dc.contributor.institutionMoscow Institute of Physics and Technology, Russia
dc.contributor.institutionUniversity of Edinburgh, United Kingdom
dc.identifier.arxivid1809.04146
kaust.personHorvath, Samuel
kaust.personRichtarik, Peter
refterms.dateFOA2019-05-29T07:43:08Z


Files in this item

Thumbnail
Name:
1809.04146.pdf
Size:
1.146Mb
Format:
PDF
Description:
Preprint

This item appears in the following Collection(s)

Show simple item record