TITLE: Integrating Augmented Reality and Artificial Intelligence for Adaptive Real-Time Task Guidance

ABSTRACT: This thesis presents an augmented reality (AR) task guidance system enhanced by artificial intelligence (AI), demonstrated through two distinct scenarios: an indoor application guiding a user to open a door handle, and an outdoor application supporting spatial navigation and interactive object identification on a university campus. Leveraging a modular architecture, the system integrates a mass-market AR headset with an AI backend that employs real-time object detection, 3D spatial tracking, and conversational interaction powered by a large language model. A key contribution of this work lies in the system-level design of a flexible AR task guidance architecture that integrates perception, tracking, navigation, and natural language interaction, and supports multiple deployment modes across edge and cloud resources. The system supports on-edge (offline) and hybrid deployment configurations, enabling the same task guidance pipeline to adapt to connectivity constraints, computational availability, and interaction requirements without modification of the core design. Further, a quantitative system-level empirical evaluation of the system consistently shows processing times below 100 ms for simultaneous tracking of multiple objects, visualization latency within acceptable real-time thresholds, and positional accuracy within centimeters in indoor scenarios. This research work establishes a measured performance baseline and provides a scalable, adaptable framework for low-latency, context-aware AR task guidance suitable for diverse real-world applications.

MAJOR ADVISOR: Raffaele De Amicis
COMMITTEE: Mike Bailey
COMMITTEE: Eugene Zhang
COMMITTEE: Stefan Lee
MINOR ADVISOR: Stefan Lee
GCR: Leonard Coop