BILGO: Bilateral greedy optimization for large scale semidefinite programming

Handle URI:
http://hdl.handle.net/10754/556165
Title:
BILGO: Bilateral greedy optimization for large scale semidefinite programming
Authors:
Hao, Zhifeng; Yuan, Ganzhao; Ghanem, Bernard ( 0000-0002-5534-587X )
Abstract:
Many machine learning tasks (e.g. metric and manifold learning problems) can be formulated as convex semidefinite programs. To enable the application of these tasks on a large-scale, scalability and computational efficiency are considered as desirable properties for a practical semidefinite programming algorithm. In this paper, we theoretically analyze a new bilateral greedy optimization (denoted BILGO) strategy in solving general semidefinite programs on large-scale datasets. As compared to existing methods, BILGO employs a bilateral search strategy during each optimization iteration. In such an iteration, the current semidefinite matrix solution is updated as a bilateral linear combination of the previous solution and a suitable rank-1 matrix, which can be efficiently computed from the leading eigenvector of the descent direction at this iteration. By optimizing for the coefficients of the bilateral combination, BILGO reduces the cost function in every iteration until the KKT conditions are fully satisfied, thus, it tends to converge to a global optimum. In fact, we prove that BILGO converges to the global optimal solution at a rate of O(1/k), where k is the iteration counter. The algorithm thus successfully combines the efficiency of conventional rank-1 update algorithms and the effectiveness of gradient descent. Moreover, BILGO can be easily extended to handle low rank constraints. To validate the effectiveness and efficiency of BILGO, we apply it to two important machine learning tasks, namely Mahalanobis metric learning and maximum variance unfolding. Extensive experimental results clearly demonstrate that BILGO can solve large-scale semidefinite programs efficiently.
KAUST Department:
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Citation:
BILGO: Bilateral greedy optimization for large scale semidefinite programming 2014, 127:247 Neurocomputing
Publisher:
Elsevier BV
Journal:
Neurocomputing
Issue Date:
3-Oct-2013
DOI:
10.1016/j.neucom.2013.07.024
Type:
Article
ISSN:
09252312
Additional Links:
http://linkinghub.elsevier.com/retrieve/pii/S0925231213007686; http://vcc.kaust.edu.sa/Documents/B.%20Ghanem/papers/BILGO_Neurocomputing2013.pdf
Appears in Collections:
Articles; Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

Full metadata record

DC FieldValue Language
dc.contributor.authorHao, Zhifengen
dc.contributor.authorYuan, Ganzhaoen
dc.contributor.authorGhanem, Bernarden
dc.date.accessioned2015-06-02T14:07:53Zen
dc.date.available2015-06-02T14:07:53Zen
dc.date.issued2013-10-03en
dc.identifier.citationBILGO: Bilateral greedy optimization for large scale semidefinite programming 2014, 127:247 Neurocomputingen
dc.identifier.issn09252312en
dc.identifier.doi10.1016/j.neucom.2013.07.024en
dc.identifier.urihttp://hdl.handle.net/10754/556165en
dc.description.abstractMany machine learning tasks (e.g. metric and manifold learning problems) can be formulated as convex semidefinite programs. To enable the application of these tasks on a large-scale, scalability and computational efficiency are considered as desirable properties for a practical semidefinite programming algorithm. In this paper, we theoretically analyze a new bilateral greedy optimization (denoted BILGO) strategy in solving general semidefinite programs on large-scale datasets. As compared to existing methods, BILGO employs a bilateral search strategy during each optimization iteration. In such an iteration, the current semidefinite matrix solution is updated as a bilateral linear combination of the previous solution and a suitable rank-1 matrix, which can be efficiently computed from the leading eigenvector of the descent direction at this iteration. By optimizing for the coefficients of the bilateral combination, BILGO reduces the cost function in every iteration until the KKT conditions are fully satisfied, thus, it tends to converge to a global optimum. In fact, we prove that BILGO converges to the global optimal solution at a rate of O(1/k), where k is the iteration counter. The algorithm thus successfully combines the efficiency of conventional rank-1 update algorithms and the effectiveness of gradient descent. Moreover, BILGO can be easily extended to handle low rank constraints. To validate the effectiveness and efficiency of BILGO, we apply it to two important machine learning tasks, namely Mahalanobis metric learning and maximum variance unfolding. Extensive experimental results clearly demonstrate that BILGO can solve large-scale semidefinite programs efficiently.en
dc.publisherElsevier BVen
dc.relation.urlhttp://linkinghub.elsevier.com/retrieve/pii/S0925231213007686en
dc.relation.urlhttp://vcc.kaust.edu.sa/Documents/B.%20Ghanem/papers/BILGO_Neurocomputing2013.pdfen
dc.rightsNOTICE: this is the author’s version of a work that was accepted for publication in Neurocomputing. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Neurocomputing, 3 October 2013. DOI: 10.1016/j.neucom.2013.07.024en
dc.subjectMetric learningen
dc.subjectLeading eigenvectoren
dc.subjectFrank–Wolfe algorithmen
dc.subjectRank-1 approximationen
dc.subjectLow-rank optimizationen
dc.subjectSemidefinite programmingen
dc.titleBILGO: Bilateral greedy optimization for large scale semidefinite programmingen
dc.typeArticleen
dc.contributor.departmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Divisionen
dc.identifier.journalNeurocomputingen
dc.eprint.versionPost-printen
dc.contributor.institutionSchool of Computer Science & Engineering, South China University of Technology, Chinaen
dc.contributor.institutionFaculty of Computer, Guangdong University of Technology, Chinaen
kaust.authorGhanem, Bernarden
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.