Can We See More? Joint Frontalization and Hallucination of Unaligned Tiny Faces
KAUST DepartmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Electrical Engineering Program
Online Publication Date2019-05-02
Print Publication Date2020-09-01
Permanent link to this recordhttp://hdl.handle.net/10754/655902
MetadataShow full item record
AbstractIn popular TV programs (such as CSI), a very low-resolution face image of a person, who is not even looking at the camera in many cases, is digitally super-resolved to a degree that suddenly the person's identity is made visible and recognizable. Of course, we suspect that this is merely a cinematographic special effect and such a magical transformation of a single image is not technically possible. Or, is it? In this paper, we push the boundaries of super-resolving (hallucinating to be more accurate) a tiny, non-frontal face image to understand how much of this is possible by leveraging the availability of large datasets and deep networks. To this end, we introduce a novel Transformative Adversarial Neural Network (TANN) to jointly frontalize very-low resolution (i.e. $16\times 16$ pixels) out-of-plane rotated face images (including profile views) and aggressively super-resolve them ($8\times$), regardless of their original poses and without using any 3D information. TANN is composed of two components: a transformative upsampling network which embodies encoding, spatial transformation and deconvolutional layers, and a discriminative network that enforces the generated high-resolution frontal faces to lie on the same manifold as real frontal face images. We evaluate our method on a large set of synthesized non-frontal face images to assess its reconstruction performance. Extensive experiments demonstrate that TANN generates both qualitatively and quantitatively superior results achieving over 4 dB improvement over the state-of-the-art.
CitationYu, X., Shiri, F., Ghanem, B., & Porikli, F. (2020). Can We See More? Joint Frontalization and Hallucination of Unaligned Tiny Faces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(9), 2148–2164. doi:10.1109/tpami.2019.2914039
SponsorsThis work was supported by the Australian Research Council‘s Discovery Projects funding scheme (project DP150104645), Australian Research Council Centre of Excellence for Robotic Vision (project number CE140100016) and King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research.