MIRROR LogoMIRROR

Visual Motion Imitation via Real-time Retargeting and TeleOpeRation with Parallel Differential Inverse Kinematics

AMBER LAB, California Institute of Technology

This work is supported by:

Visual Skeleton Motion Capture, Retargeting and, Teleoperation on Humanoid Robot.

Abstract

Real-time humanoid teleoperation requires inverse kinematics (IK) solvers that are both responsive and constraint-safe under kinematic redundancy and self-collision constraints. While differential IK enables efficient online retargeting, its locally linearized updates are inherently basin-dependent and often become trapped near joint limits, singularities, or active collision boundaries, leading to unsafe or stagnant behavior. We propose a GPU-parallelized, continuation-based differential IK that improves escape from such constraint-induced local minima while preserving real-time performance, promoting safety and stability. Multiple constrained IK quadratic programs are evaluated in parallel, together with a self-collision avoidance control barrier function (CBF), and a Lyapunov-based progression criterion selects updates that reduce the final global task-space error. The method is paired with a visual skeletal pose estimation pipeline that enables robust, real-time upper-body teleoperation on the THEMIS humanoid robot hardware in real-world tasks.

Full Supplementary Video

System Architecture

Video Gallery

Self-Collision Avoidance Demonstration

Joint Stagnation Avoidance Demonstration

Robustness to Multi-Target Perturbations

Partial Occlusion Handling Demonstration

Teleoperating Object Removal

Teleoperating Object Transfer

Direct PD-Control Mode

WBC Mode

🤖 Catch the Blue Can Challenge

Score: 0 | High Score: 0

←/→ Move arm | SPACE Catch! | R Restart