Guess where? Actor-supervision for spatiotemporal action localization

Embargo End Date
2021-12-09

Type
Article

Authors
Escorcia, Victor
Dao, Cuong D.
Jain, Mihir
Ghanem, Bernard
Snoek, Cees

KAUST Department
Electrical Engineering Program
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division

KAUST Grant Number
OSR-CRG2017-3405

Preprint Posting Date
2018-04-05

Online Publication Date
2019-12-09

Print Publication Date
2020-03

Date
2019-12-09

Submitted Date
2018-12-19

Abstract
This paper addresses the problem of spatiotemporal localization of actions in videos. Compared to leading approaches, which all learn to localize based on carefully annotated boxes on training video frames, we adhere to a solution only requiring video class labels. We introduce an actor-supervised architecture that exploits the inherent compositionality of actions in terms of actor transformations, to localize actions. We make two contributions. First, we propose actor proposals derived from a detector for human and non-human actors intended for images, which are linked over time by Siamese similarity matching to account for actor deformations. Second, we propose an actor-based attention mechanism enabling localization from action class labels and actor proposals. It exploits a new actor pooling operation and is end-to-end trainable. Experiments on four action datasets show actor supervision is state-of-the-art for action localization from video class labels and is even competitive to some box-supervised alternatives.

Citation
Escorcia, V., Dao, C. D., Jain, M., Ghanem, B., & Snoek, C. (2020). Guess where? Actor-supervision for spatiotemporal action localization. Computer Vision and Image Understanding, 192, 102886. doi:10.1016/j.cviu.2019.102886

Acknowledgements
This publication is based upon work supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research (OSR) under Award No. OSR-CRG2017-3405. We thank the team members of the IVUL from KAUST and Qualcomm AI Research for helpful comments and discussion. In particular, we appreciate the support of Amirhossein Habibian during the implementation of our Actor Linking.

Publisher
Elsevier BV

Journal
Computer Vision and Image Understanding

DOI
10.1016/j.cviu.2019.102886

arXiv
1804.01824

Additional Links
https://linkinghub.elsevier.com/retrieve/pii/S1077314219301687

Permanent link to this record
Collections

Version History

Now showing 1 - 2 of 2
VersionDateSummary
2*
2020-03-16 05:17:26
Updated with accepted manuscript version
2018-04-16 11:27:42
* Selected version