Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization
KAUST DepartmentComputer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division
Computer Science Program
Electrical Engineering Program
Visual Computing Center (VCC)
KAUST Grant NumberOSR-CRG2017-3405
Online Publication Date2018-10-05
Print Publication Date2018
Permanent link to this recordhttp://hdl.handle.net/10754/630233
MetadataShow full item record
AbstractState-of-the-art temporal action detectors inefficiently search the entire video for specific actions. Despite the encouraging progress these methods achieve, it is crucial to design automated approaches that only explore parts of the video which are the most relevant to the actions being searched for. To address this need, we propose the new problem of action spotting in video, which we define as finding a specific action in a video while observing a small portion of that video. Inspired by the observation that humans are extremely efficient and accurate in spotting and finding action instances in video, we propose Action Search, a novel Recurrent Neural Network approach that mimics the way humans spot actions. Moreover, to address the absence of data recording the behavior of human annotators, we put forward the Human Searches dataset, which compiles the search sequences employed by human annotators spotting actions in the AVA and THUMOS14 datasets. We consider temporal action localization as an application of the action spotting problem. Experiments on the THUMOS14 dataset reveal that our model is not only able to explore the video efficiently (observing on average 17.3% of the video) but it also accurately finds human activities with 30.8% mAP.
CitationAlwassel H, Caba Heilbron F, Ghanem B (2018) Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization. Lecture Notes in Computer Science: 253–269. Available: http://dx.doi.org/10.1007/978-3-030-01240-3_16.
SponsorsThis publication is based upon work supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research (OSR) under Award No. OSR-CRG2017-3405.
Conference/Event name15th European Conference on Computer Vision, ECCV 2018
Showing items related by title, author, creator and subject.
DAPs: Deep Action Proposals for Action UnderstandingEscorcia, Victor; Caba Heilbron, Fabian; Niebles, Juan Carlos; Ghanem, Bernard (Lecture Notes in Computer Science, Springer Nature, 2016-09-17) [Conference Paper]Object proposals have contributed significantly to recent advances in object understanding in images. Inspired by the success of this approach, we introduce Deep Action Proposals (DAPs), an effective and efficient algorithm for generating temporal action proposals from long videos. We show how to take advantage of the vast capacity of deep learning models and memory cells to retrieve from untrimmed videos temporal segments, which are likely to contain actions. A comprehensive evaluation indicates that our approach outperforms previous work on a large scale action benchmark, runs at 134 FPS making it practical for large-scale scenarios, and exhibits an appealing ability to generalize, i.e. to retrieve good quality temporal proposals of actions unseen in training.
On the feasibility of inducing oil mobilization in existing reservoirs via wellbore harmonic fluid actionJeong, Chanseok; Huh, Chun; Kallivokas, Loukas F. (Journal of Petroleum Science and Engineering, Elsevier BV, 2011-03) [Article]Although vibration-based mobilization of oil remaining in mature reservoirs is a promising low-cost method of enhanced oil recovery (EOR), research on its applicability at the reservoir scale is still at an early stage. In this paper, we use simplified models to study the potential for oil mobilization in homogeneous and fractured reservoirs, when harmonically oscillating fluids are injected/produced within a well. To this end, we investigate first whether waves, induced by fluid pressure oscillations at the well site, and propagating radially and away from the source in a homogeneous reservoir, could lead to oil droplet mobilization in the reservoir pore-space. We discuss both the fluid pore-pressure wave and the matrix elastic wave cases, as potential agents for increasing oil mobility. We then discuss the more realistic case of a fractured reservoir, where we study the fluid pore-pressure wave motion, while taking into account the leakage effect on the fracture wall. Numerical results show that, in homogeneous reservoirs, the rock-stress wave is a better energy-delivery agent than the fluid pore-pressure wave. However, neither the rock-stress wave nor the pore-pressure wave is likely to result in any significant residual oil mobilization at the reservoir scale. On the other hand, enhanced oil production from the fractured reservoir's matrix zone, induced by cross-flow vibrations, appears to be feasible. In the fractured reservoir, the fluid pore-pressure wave is only weakly attenuated through the fractures, and thus could induce fluid exchange between the rock formation and the fracture space. The vibration-induced cross-flow is likely to improve the imbibition of water into the matrix zone and the expulsion of oil from it. © 2011 Elsevier B.V.
The Mode of Action of Isocyanide in Three Aquatic Organisms, Balanus amphitrite, Bugula neritina and Danio rerioZhang, Yi-Fan; Kitano, Yoshikazu; Nogata, Yasuyuki; Zhang, Yu; Qian, Pei-Yuan (PLoS ONE, Public Library of Science (PLoS), 2012-09-18) [Article]Isocyanide is a potential antifouling compound in marine environments. In this study, we investigated its mode of action in three aquatic organisms. Two of them, the bryozoan Bugula neritina and the barnacle Balanus amphitrite, are major marine fouling invertebrates, and the other organism is the non-target species zebrafish Danio rerio. In the swimming larvae of B. neritina, isocyanide did not affect the total attachment rate (≤50 µg ml^(−1)), but it did change the attachment site by increasing the percentage of attachment on the bottom of the container rather than on the wall or air-water inter-surface. Isocyanide binds several proteins in B. neritina as identified via SDS-PAGE-LC-MS/MS: 1) a 30 kD protein band containing two proteins similar to voltage dependent anion channels (VDAC), which control the direct coupling of the mitochondrial matrix to the energy maintenance of the cytosol and the release of apoptogenic factors from mitochondria of mammalian cells; and 2) an unknown 39 kD protein. In B. amphitrite cyprids, the isocyanide binding protein were 1) a protein similar to NADH-ubiquinone oxidoreductase, which is the “entry enzyme” of oxidative phosphorylation in mitochondria; and 2) cytochrome P450. In Danio rerio embryos, isocyanide caused “wavy” notochords, hydrocephalus, pericardial edema, poor blood circulation, and defects in pigmentation and hematopoiesis, which phenocopied copper deficiency. This is the first report on isocyanide binding proteins in fouling organisms, as well as the first description of its phenotype and potential toxicology in zebrafish.