Skip to main content

VR Teleoperation

Meta Quest hand and wrist tracking drives both arms of the bimanual rig in real time — a browser-side WebXR page streams hand poses over a secure WebSocket to a C++ ROS 2 bridge, which feeds a per-arm damped-least-squares IK controller in Isaac Sim, with a stereo POV feed rendered back to the headset so the operator sees through the robot's own cameras. The coolest part: it isn't sim-only. A gated flag streams the left arm's live IK solution straight to the physical robot over CAN, so the same pinch-to-grip, wrist-tracked teleop that drives the simulated arm can drive the real hardware in lockstep — headset in, robot arm moving, all over a USB tunnel with no Wi-Fi setup required.

Controls​

ActionEffect
Move right / left wristCorresponding arm follows
Pinch thumb + index (either hand)Corresponding gripper closes / opens
R (Isaac Sim window focused)Recalibrate — re-home both arms to current wrist pose
--gain VALUE (default 1.0)Scales arm motion per metre of real wrist motion