Skip to main content

Reinforcement Learning

In-Hand Manipulation RL Task​

Reinforcement learning-based in-hand cube reorientation task for the 16 DOF Wato hand in Isaac Lab. The policy learns to reorient a DexCube held in the palm toward commanded goal orientations using PPO (Proximal Policy Optimization).

Task Setup​

ParameterValue
Robot16 DOF Wato hand (palm-up orientation)
ObjectIsaac Nucleus DexCube, scale (0.8, 0.8, 0.8)
Goal commandReorientation command, resampled on success
Success thresholdOrientation error < 0.4 rad
Episode length20 seconds
Number of environments2048 (default)
Simulation frequency120 Hz
Action decimation4

Reward Structure​

The reward function combines task objectives with regularization penalties:

CategoryTermWeightDescription
Task RewardsPosition tracking-3.0L2 distance to goal position (hold in palm)
Orientation tracking10.0Dense rotation signal: 1/(error+0.1)1 / (\text{error} + 0.1)
Success bonus50.0Binary reward when error < 0.4 rad
Object held bonus0.5+1/step when cube within 0.10 m of goal
Angular velocity toward goal0.2Reward cube spin aligned with goal
Spread activity0.03Encourages finger abduction (Wato hand)
PenaltiesObject away penalty-5.0Terminal penalty when out of reach
Action rate L2-0.05Penalize jerky commands
Joint velocity L2−1×10−4-1 \times 10^{-4}Penalize high joint speeds
Action L2−1×10−4-1 \times 10^{-4}Penalize large action magnitudes

Termination Conditions​

ConditionThreshold
Time outEpisode length > 20 seconds
Max consecutive success50 successful reorientations in one episode
Object out of reachCube drifts > 0.3 m from robot root
Orientation stagnationError stays > 0.5 rad for 150 consecutive steps

Training Details​

  • Algorithm: PPO with GAE (Generalized Advantage Estimation)
  • Gamma: 0.998
  • Entropy coefficient: 0.0001
  • Steps per environment: 48
  • Action smoothing: EMA joint-position targets, alpha = 0.85
  • Max iterations: 5000

The task is implemented in Isaac Lab and adapted from their in-hand manipulation examples. Checkpoints are saved to logs/rsl_rl/wato_hand_cube/.

Additional RL Demonstrations​

Manipulation Task Setup​

Parallelized manipulation task environments for RL training.

Manipulation task setup, parallel environments