Facial Action Unit Recognition under Incomplete Data Based on Multi-label Learning with Missing Labels

Handle URI:
http://hdl.handle.net/10754/615897
Title:
Facial Action Unit Recognition under Incomplete Data Based on Multi-label Learning with Missing Labels
Authors:
Li, Yongqiang; Wu, Baoyuan ( 0000-0003-2183-5990 ) ; Ghanem, Bernard ( 0000-0002-5534-587X ) ; Zhao, Yongping; Yao, Hongxun; Ji, Qiang
Abstract:
Facial action unit (AU) recognition has been applied in a wild range of fields, and has attracted great attention in the past two decades. Most existing works on AU recognition assumed that the complete label assignment for each training image is available, which is often not the case in practice. Labeling AU is expensive and time consuming process. Moreover, due to the AU ambiguity and subjective difference, some AUs are difficult to label reliably and confidently. Many AU recognition works try to train the classifier for each AU independently, which is of high computation cost and ignores the dependency among different AUs. In this work, we formulate AU recognition under incomplete data as a multi-label learning with missing labels (MLML) problem. Most existing MLML methods usually employ the same features for all classes. However, we find this setting is unreasonable in AU recognition, as the occurrence of different AUs produce changes of skin surface displacement or face appearance in different face regions. If using the shared features for all AUs, much noise will be involved due to the occurrence of other AUs. Consequently, the changes of the specific AUs cannot be clearly highlighted, leading to the performance degradation. Instead, we propose to extract the most discriminative features for each AU individually, which are learned by the supervised learning method. The learned features are further embedded into the instance-level label smoothness term of our model, which also includes the label consistency and the class-level label smoothness. Both a global solution using st-cut and an approximated solution using conjugate gradient (CG) descent are provided. Experiments on both posed and spontaneous facial expression databases demonstrate the superiority of the proposed method in comparison with several state-of-the-art works.
KAUST Department:
Visual Computing Center (VCC)
Citation:
Facial Action Unit Recognition under Incomplete Data Based on Multi-label Learning with Missing Labels 2016 Pattern Recognition
Publisher:
Elsevier BV
Journal:
Pattern Recognition
Issue Date:
7-Jul-2016
DOI:
10.1016/j.patcog.2016.07.009
Type:
Article
ISSN:
00313203
Sponsors:
Yongqiang Li is supported by National Natural Science Foundation of China (No. 61402129), and Postdoctoral Foundation Projects (No. LBH-Z14090 and No. 2015M571417). Bernard Ghanem and Baoyuan Wu are supported by funding from King Abdullah University of Science and Technology (KAUST). Hongxun Yao is partially supported by National Natural Science Foundation of China (No. 61472103) and Key Program (No. 61133003).
Additional Links:
http://linkinghub.elsevier.com/retrieve/pii/S0031320316301583
Appears in Collections:
Articles

Full metadata record

DC FieldValue Language
dc.contributor.authorLi, Yongqiangen
dc.contributor.authorWu, Baoyuanen
dc.contributor.authorGhanem, Bernarden
dc.contributor.authorZhao, Yongpingen
dc.contributor.authorYao, Hongxunen
dc.contributor.authorJi, Qiangen
dc.date.accessioned2016-07-11T09:32:51Z-
dc.date.available2016-07-11T09:32:51Z-
dc.date.issued2016-07-07-
dc.identifier.citationFacial Action Unit Recognition under Incomplete Data Based on Multi-label Learning with Missing Labels 2016 Pattern Recognitionen
dc.identifier.issn00313203-
dc.identifier.doi10.1016/j.patcog.2016.07.009-
dc.identifier.urihttp://hdl.handle.net/10754/615897-
dc.description.abstractFacial action unit (AU) recognition has been applied in a wild range of fields, and has attracted great attention in the past two decades. Most existing works on AU recognition assumed that the complete label assignment for each training image is available, which is often not the case in practice. Labeling AU is expensive and time consuming process. Moreover, due to the AU ambiguity and subjective difference, some AUs are difficult to label reliably and confidently. Many AU recognition works try to train the classifier for each AU independently, which is of high computation cost and ignores the dependency among different AUs. In this work, we formulate AU recognition under incomplete data as a multi-label learning with missing labels (MLML) problem. Most existing MLML methods usually employ the same features for all classes. However, we find this setting is unreasonable in AU recognition, as the occurrence of different AUs produce changes of skin surface displacement or face appearance in different face regions. If using the shared features for all AUs, much noise will be involved due to the occurrence of other AUs. Consequently, the changes of the specific AUs cannot be clearly highlighted, leading to the performance degradation. Instead, we propose to extract the most discriminative features for each AU individually, which are learned by the supervised learning method. The learned features are further embedded into the instance-level label smoothness term of our model, which also includes the label consistency and the class-level label smoothness. Both a global solution using st-cut and an approximated solution using conjugate gradient (CG) descent are provided. Experiments on both posed and spontaneous facial expression databases demonstrate the superiority of the proposed method in comparison with several state-of-the-art works.en
dc.description.sponsorshipYongqiang Li is supported by National Natural Science Foundation of China (No. 61402129), and Postdoctoral Foundation Projects (No. LBH-Z14090 and No. 2015M571417). Bernard Ghanem and Baoyuan Wu are supported by funding from King Abdullah University of Science and Technology (KAUST). Hongxun Yao is partially supported by National Natural Science Foundation of China (No. 61472103) and Key Program (No. 61133003).en
dc.language.isoenen
dc.publisherElsevier BVen
dc.relation.urlhttp://linkinghub.elsevier.com/retrieve/pii/S0031320316301583en
dc.rightsNOTICE: this is the author’s version of a work that was accepted for publication in Pattern Recognition. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Pattern Recognition, 7 July 2016. DOI: 10.1016/j.patcog.2016.07.009en
dc.subjectFace action unit recognitionen
dc.subjectIncomplete dataen
dc.subjectMulti-label learningen
dc.titleFacial Action Unit Recognition under Incomplete Data Based on Multi-label Learning with Missing Labelsen
dc.typeArticleen
dc.contributor.departmentVisual Computing Center (VCC)en
dc.identifier.journalPattern Recognitionen
dc.eprint.versionPost-printen
dc.contributor.institutionSchool of Electrical Engineering and Automation, Harbin Institute of Technology, Harbin 15001, Chinaen
dc.contributor.institutionSchool of Computer Science and Technology, Harbin Institute of Technology, Harbin 15001, Chinaen
dc.contributor.institutionDepartment of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USAen
dc.contributor.affiliationKing Abdullah University of Science and Technology (KAUST)en
kaust.authorWu, Baoyuanen
kaust.authorGhanem, Bernarden
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.