An Efficient Multimodal 2D + 3D Feature-based Approach to Automatic Facial Expression Recognition

Handle URI:
http://hdl.handle.net/10754/561399
Title:
An Efficient Multimodal 2D + 3D Feature-based Approach to Automatic Facial Expression Recognition
Authors:
Li, Huibin; Ding, Huaxiong; Huang, Di; Wang, Yunhong; Zhao, Xi; Morvan, Jean-Marie; Chen, Liming
Abstract:
We present a fully automatic multimodal 2D + 3D feature-based facial expression recognition approach and demonstrate its performance on the BU-3DFE database. Our approach combines multi-order gradient-based local texture and shape descriptors in order to achieve efficiency and robustness. First, a large set of fiducial facial landmarks of 2D face images along with their 3D face scans are localized using a novel algorithm namely incremental Parallel Cascade of Linear Regression (iPar-CLR). Then, a novel Histogram of Second Order Gradients (HSOG) based local image descriptor in conjunction with the widely used first-order gradient based SIFT descriptor are used to describe the local texture around each 2D landmark. Similarly, the local geometry around each 3D landmark is described by two novel local shape descriptors constructed using the first-order and the second-order surface differential geometry quantities, i.e., Histogram of mesh Gradients (meshHOG) and Histogram of mesh Shape index (curvature quantization, meshHOS). Finally, the Support Vector Machine (SVM) based recognition results of all 2D and 3D descriptors are fused at both feature-level and score-level to further improve the accuracy. Comprehensive experimental results demonstrate that there exist impressive complementary characteristics between the 2D and 3D descriptors. We use the BU-3DFE benchmark to compare our approach to the state-of-the-art ones. Our multimodal feature-based approach outperforms the others by achieving an average recognition accuracy of 86.32%. Moreover, a good generalization ability is shown on the Bosphorus database.
KAUST Department:
Visual Computing Center (VCC)
Citation:
An Efficient Multimodal 2D + 3D Feature-based Approach to Automatic Facial Expression Recognition 2015 Computer Vision and Image Understanding
Publisher:
Elsevier BV
Journal:
Computer Vision and Image Understanding
Issue Date:
29-Jul-2015
DOI:
10.1016/j.cviu.2015.07.005
Type:
Article
ISSN:
10773142
Additional Links:
http://linkinghub.elsevier.com/retrieve/pii/S1077314215001587
Appears in Collections:
Articles; Visual Computing Center (VCC)

Full metadata record

DC FieldValue Language
dc.contributor.authorLi, Huibinen
dc.contributor.authorDing, Huaxiongen
dc.contributor.authorHuang, Dien
dc.contributor.authorWang, Yunhongen
dc.contributor.authorZhao, Xien
dc.contributor.authorMorvan, Jean-Marieen
dc.contributor.authorChen, Limingen
dc.date.accessioned2015-08-02T10:24:32Zen
dc.date.available2015-08-02T10:24:32Zen
dc.date.issued2015-07-29en
dc.identifier.citationAn Efficient Multimodal 2D + 3D Feature-based Approach to Automatic Facial Expression Recognition 2015 Computer Vision and Image Understandingen
dc.identifier.issn10773142en
dc.identifier.doi10.1016/j.cviu.2015.07.005en
dc.identifier.urihttp://hdl.handle.net/10754/561399en
dc.description.abstractWe present a fully automatic multimodal 2D + 3D feature-based facial expression recognition approach and demonstrate its performance on the BU-3DFE database. Our approach combines multi-order gradient-based local texture and shape descriptors in order to achieve efficiency and robustness. First, a large set of fiducial facial landmarks of 2D face images along with their 3D face scans are localized using a novel algorithm namely incremental Parallel Cascade of Linear Regression (iPar-CLR). Then, a novel Histogram of Second Order Gradients (HSOG) based local image descriptor in conjunction with the widely used first-order gradient based SIFT descriptor are used to describe the local texture around each 2D landmark. Similarly, the local geometry around each 3D landmark is described by two novel local shape descriptors constructed using the first-order and the second-order surface differential geometry quantities, i.e., Histogram of mesh Gradients (meshHOG) and Histogram of mesh Shape index (curvature quantization, meshHOS). Finally, the Support Vector Machine (SVM) based recognition results of all 2D and 3D descriptors are fused at both feature-level and score-level to further improve the accuracy. Comprehensive experimental results demonstrate that there exist impressive complementary characteristics between the 2D and 3D descriptors. We use the BU-3DFE benchmark to compare our approach to the state-of-the-art ones. Our multimodal feature-based approach outperforms the others by achieving an average recognition accuracy of 86.32%. Moreover, a good generalization ability is shown on the Bosphorus database.en
dc.language.isoenen
dc.publisherElsevier BVen
dc.relation.urlhttp://linkinghub.elsevier.com/retrieve/pii/S1077314215001587en
dc.rightsNOTICE: this is the author’s version of a work that was accepted for publication in Computer Vision and Image Understanding. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Computer Vision and Image Understanding, 29 July 2015. DOI: 10.1016/j.cviu.2015.07.005en
dc.subjectFacial expression recognitionen
dc.subjectLocal texture descriptoren
dc.subjectLocal shape descriptoren
dc.subjectMultimodal fusionen
dc.titleAn Efficient Multimodal 2D + 3D Feature-based Approach to Automatic Facial Expression Recognitionen
dc.typeArticleen
dc.contributor.departmentVisual Computing Center (VCC)en
dc.identifier.journalComputer Vision and Image Understandingen
dc.eprint.versionPost-printen
dc.contributor.institutionSchool of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Chinaen
dc.contributor.institutionEcole Centrale de Lyon, LIRIS UMR5205, Lyon, Franceen
dc.contributor.institutionState Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, Beijing, Chinaen
dc.contributor.institutionSchool of Management, Xi’an Jiaotong University, Xi’an, Chinaen
dc.contributor.institutionUniversité Lyon 1, Institut Camille Jordan, Lyon, Franceen
dc.contributor.affiliationKing Abdullah University of Science and Technology (KAUST)en
kaust.authorMorvan, Jean-Marieen
All Items in KAUST are protected by copyright, with all rights reserved, unless otherwise indicated.