TITLE: Training Multi-Modal Whole Body Control for Humanoid Robots

ABSTRACT: A central challenge in humanoid robotics is designing interfaces through which diverse upstream systems — teleoperation, motion capture, task planners — can command whole-body behavior. Most approaches handle each modality with a separate controller, resulting in fragmented systems that are difficult to extend. This proposal argues that the design of command representations is a key lever for building versatile humanoid controllers, and develops this idea across three problems. We start with the Masked Humanoid Controller (MHC), which accepts partially-specified kinematic trajectories with binary masks over the robot's state variables. This lets a single policy handle velocity-based walking, partial-body mimicry, and full-body motion tracking through one interface. Next, we address short-range locomotion, where velocity-target commands produce inefficient marching behavior for pose-reaching tasks. Our GoTo controller operates directly on SE(2) targets using a constellation-based reward — a geometric objective that couples translational and rotational progress through the moment of inertia of a virtual point set rigidly attached to the robot's base. Finally, we extend the command interface to object interaction with OIL, an Object Interaction Layer built on top of the frozen MHC. OIL augments the existing kinematic directive with object pose targets and contact conditions, enabling a single policy to execute various interaction behaviors without task-specific retraining. All three systems are trained via reinforcement learning in simulation with domain randomization and validated through sim-to-real transfer on the Digit V3 humanoid. Taken together, these contributions demonstrate that command representation design — what information the controller receives and in what form — is central to achieving versatile, robust, and transferable humanoid whole-body control.

MAJOR ADVISOR: Alan Fern
COMMITTEE: Stefan Lee
COMMITTEE: Fuxin Li
COMMITTEE: Geoffrey Hollinger
GCR: Joshua Reeves