Show simple item record

dc.contributor.advisorKeyes, David E.
dc.contributor.authorAlnasser, Ali
dc.date.accessioned2021-02-10T08:45:01Z
dc.date.available2021-02-10T08:45:01Z
dc.date.issued2021-02-10
dc.identifier.citationAlnasser, A. (2021). An Empirical Study of the Distributed Ellipsoidal Trust Region Method for Large Batch Training. KAUST Research Repository. https://doi.org/10.25781/KAUST-3IQ6E
dc.identifier.doi10.25781/KAUST-3IQ6E
dc.identifier.urihttp://hdl.handle.net/10754/667327
dc.description.abstractNeural networks optimizers are dominated by rst-order methods, due to their inexpensive computational cost per iteration. However, it has been shown that rstorder optimization is prone to reaching sharp minima when trained with large batch sizes. As the batch size increases, the statistical stability of the problem increases, a regime that is well suited for second-order optimization methods. In this thesis, we study a distributed ellipsoidal trust region model for neural networks. We use a block diagonal approximation of the Hessian, assigning consecutive layers of the network to each process. We solve in parallel for the update direction of each subset of the parameters. We show that our optimizer is t for large batch training as well as increasing number of processes.
dc.language.isoen
dc.subjectoptimization
dc.subjecttrust region
dc.subjectdistributed computing
dc.subjectdeep learning
dc.subjectmachine learning
dc.titleAn Empirical Study of the Distributed Ellipsoidal Trust Region Method for Large Batch Training
dc.typeThesis
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
thesis.degree.grantorKing Abdullah University of Science and Technology
dc.contributor.committeememberWonka, Peter
dc.contributor.committeememberZhang, Xiangliang
thesis.degree.disciplineComputer Science
thesis.degree.nameMaster of Science
refterms.dateFOA2021-02-10T08:45:02Z
kaust.request.doiyes


Files in this item

Thumbnail
Name:
MS_Thesis_Ali_Final_Draft.pdf
Size:
4.192Mb
Format:
PDF

This item appears in the following Collection(s)

Show simple item record