Show simple item record

dc.contributor.authorDeng, Zhiwei
dc.contributor.authorNarasimhan, Karthik
dc.contributor.authorRussakovsky, Olga
dc.date.accessioned2020-07-27T12:59:21Z
dc.date.available2020-07-27T12:59:21Z
dc.date.issued2020-01-01
dc.identifier.issn1049-5258
dc.identifier.urihttp://hdl.handle.net/10754/664432
dc.description.abstractThe ability to perform effective planning is crucial for building an instruction-following agent. When navigating through a new environment, an agent is challenged with (1) connecting the natural language instructions with its progressively growing knowledge of the world; and (2) performing long-range planning and decision making in the form of effective exploration and error correction. Current methods are still limited on both fronts despite extensive efforts. In this paper, we introduce the Evolving Graphical Planner (EGP), a model that performs global planning for navigation based on raw sensory input. The model dynamically constructs a graphical representation, generalizes the action space to allow for more flexible decision making, and performs efficient planning on a proxy graph representation. We evaluate our model on a challenging Vision-and-Language Navigation (VLN) task with photorealistic images, and achieve superior performance compared to previous navigation architectures. For instance, we achieve a 53% success rate on the test split of the Room-to-Room navigation task [1] through pure imitation learning, outperforming previous navigation architectures by up to 5%.
dc.description.sponsorshipThis work is partially supported by King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research (OSR) under Award No. OSRCRG2017-3405 and by Princeton University’s Center for Statistics and Machine Learning (CSML) DataX fund. We would also like to thank Felix Yu, Angelina Wang and Zeyu Wang for offering insightful discussions and comments on the paper.
dc.publisherNeural information processing systems foundation
dc.relation.urlhttps://proceedings.neurips.cc/paper/2020/hash/eddb904a6db773755d2857aacadb1cb0-Abstract.html
dc.rightsArchived with thanks to Neural information processing systems foundation
dc.titleEvolving graphical planner: Contextual global planning for vision-and-language navigation
dc.typeConference Paper
dc.conference.date2020-12-06 to 2020-12-12
dc.conference.name34th Conference on Neural Information Processing Systems, NeurIPS 2020
dc.conference.locationVirtual, Online
dc.eprint.versionPre-print
dc.contributor.institutionDepartment of Computer Science Princeton University
dc.identifier.volume2020-December
dc.identifier.arxivid2007.05655
kaust.grant.numberOSRCRG2017-3405
dc.identifier.eid2-s2.0-85107984383
refterms.dateFOA2020-07-27T13:00:04Z
kaust.acknowledged.supportUnitOffice of Sponsored Research (OSR)


Files in this item

Thumbnail
Name:
Preprintfile1.pdf
Size:
5.672Mb
Format:
PDF
Description:
Pre-print

This item appears in the following Collection(s)

Show simple item record