Semantic Embodied Navigation: Developing Agents That Navigate From Language and Vision

Autonomous robotic agents are on their way to becoming in-home personal assistants, construction assistants, and warehouse workers. The degree of autonomy of such systems is reflected by the manner in which we specify our goals to them; the abstraction of low-level commands to high-level goals goes hand-in-hand with increased autonomy. In this work, we are interested in developing artificial intelligence that enables agents to accomplish semantic goals. We specifically study semantic embodied navigation. In embodied navigation, agents act in spatial environments to follow a motion path or reach a destination. The qualifier semantic refers to how the goal is specified; instead of directly (e.g. coordinates), a goal is provided in a manner that either implies the goal or constrains the possible interpretations of the goal. We study two abilities of semantic embodied navigation: navigating in response to natural language instructions and navigating to an object specified by an image. Our contributions are aimed at developing fundamental semantic-guided navigation abilities for embodied agents and are highlighted as follows. For both tasks, we establish benchmarks in simulation and evaluate baseline models. For language-based navigation, we develop language-conditioned waypoint prediction networks to study the impact of mid-level action spaces. We then show that by transferring an agent from highly-abstract simulation to lower-level simulation, we can achieve high performance on our language navigation benchmark. For image-based navigation, we find that existing methods that train sensors-to-action policies struggle to generalize to new environments and new goals. We propose decomposing the image navigation task into sub-tasks addressable via modular components. We will evaluate the resulting agent both on our simulation benchmark and demonstrate performance in the real world.

MAJOR ADVISOR: Stefan Lee
COMMITTEE: Prasad Tadepalli
COMMITTEE: Alan Fern
COMMITTEE: Fuxin Li
GCR: Joseph Louis