Type
Conference PaperKAUST Grant Number
CRG-2017-3426BAS/1/1685-01-01
Date
2021-11-02Preprint Posting Date
2021-01-19Online Publication Date
2021-11-02Print Publication Date
2021-06Permanent link to this record
http://hdl.handle.net/10754/668789
Metadata
Show full item recordAbstract
We present a novel large-scale dataset and accompanying machine learning models aimed at providing a detailed understanding of the interplay between visual content, its emotional effect, and explanations for the latter in language. In contrast to most existing annotation datasets in computer vision, we focus on the affective experience triggered by visual artworks and ask the annotators to indicate the dominant emotion they feel for a given image and, crucially, to also provide a grounded verbal explanation for their emotion choice. As we demonstrate below, this leads to a rich set of signals for both the objective content and the affective impact of an image, creating associations with abstract concepts (e.g., "freedom" or "love"), or references that go beyond what is directly visible, including visual similes and metaphors, or subjective references to personal experiences. We focus on visual art (e.g., paintings, artistic photographs) as it is a prime example of imagery created to elicit emotional responses from its viewers. Our dataset, termed ArtEmis, contains 455K emotion attributions and explanations from humans, on 80K artworks from WikiArt. Building on this data, we train and demonstrate a series of captioning systems capable of expressing and explaining emotions from visual stimuli. Remarkably, the captions produced by these systems often succeed in reflecting the semantic and abstract content of the image, going well beyond systems trained on existing datasets. The collected dataset and developed methods are available at https://artemisdataset.org.Citation
Achlioptas, P., Ovsjanikov, M., Haydarov, K., Elhoseiny, M., & Guibas, L. (2021). ArtEmis: Affective Language for Visual Art. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). doi:10.1109/cvpr46437.2021.01140Sponsors
This work is funded by a Vannevar Bush Faculty Fellowship, a KAUST BAS/1/1685-01-01, a CRG-2017-3426, the ERC Starting Grant No. 758800 (EXPROTEA) and the ANR AI Chair AIGRETTE, and gifts from the Adobe, Amazon AWS, Autodesk, and Snap corporations. The authors wish to thank Fei Xia and Jan Dombrowski for their help with the AMT instruction design and Nikos Gkanatsios for several fruitful discussions. The authors also want to emphasize their gratitude to all the hard working Amazon Mechanical Turkers without whom this work would not be possible.Publisher
IEEEConference/Event name
2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)ISBN
978-1-6654-4510-8arXiv
2101.07396Additional Links
https://ieeexplore.ieee.org/document/9577962/https://ieeexplore.ieee.org/document/9577962/
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9577962
ae974a485f413a2113503eed53cd6c53
10.1109/CVPR46437.2021.01140