Humanoid table tennis
Teaching a humanoid to return a table-tennis ball takes more than a fast swing. HOPE connects motion-capture perception, physics-based racket planning, and learned whole-body control so the Agibot A3 can coordinate a strike, maintain balance, and recover for the next ball.
ROS 2Isaac Lab / PPOMuJoCoA closer lookShow less
A complete perception-to-action loop
- 01ObserveTrack the ball
- 02PlanChoose the strike
- 03ActMove the whole body
- 04RecoverPrepare for the next ball
The planner decides where and when the racket should meet the ball. A learned controller turns that compact goal into coordinated joint motion. This division connects explicit ball physics with a policy that handles the robot’s body.
The challenge: timing, contact, and balance
A useful return needs the right contact position, racket velocity, and strike time. The robot must also stay upright and finish in a state from which it can strike again. HOPE brings these pieces together in an open-source stack for the Agibot A3, with a unified forehand/backhand policy and a shared path from training to deployment.
Tracking the ball and planning a return
OptiTrack/NatNet or Chingmu/VRPN supplies motion-capture measurements through ROS 2. A C++ planner estimates the incoming flight and predicts gravity, aerodynamic drag, and table bounces. The no-spin physics model uses parameters fitted from real venue measurements.
At a virtual hitting plane, the planner chooses forehand or backhand, solves for a return toward the opponent’s half, and produces racket position, velocity, face normal, and timing. One frozen flight packet is used for each solve; the runtime holds the accepted command once the swing engages, keeping the target stable through the strike.
Planner architectureOne policy for both sides
A single feed-forward policy runs at 50 Hz for both forehand and backhand. It observes joint state, body motion and orientation, the offset to its recovery station, and the racket’s target position, velocity, and time-to-strike. Ball trajectories and stroke selection are handled outside the actor.
- Observation
- 110 dimensions
- Action
- 31 joint residuals
- Control
- 50 Hz
The outputs become joint-position targets under the A3’s joint limits; the runtime keeps the two head joints at their nominal pose. The policy’s station feedback supports in-place recovery between strokes.
Learning a continuous rally
Training uses PPO in Isaac Lab with a forehand/backhand reference-motion pair. A privileged critic can use reference motion and true simulated racket state during training, while the deployed actor uses only its own observation contract.
Robot state carries across successive swings instead of resetting after every shot. Training therefore includes forehand-to-backhand transitions, balance, and recovery as part of a continuous sequence. Racket goals include samples fitted to the real venue; mass variation and external pushes add physical variation. Control runs at 50 Hz over 200 Hz simulated physics.
Training designKeeping simulation and deployment aligned
The trained actor is exported to ONNX with metadata for observation layout, joint order, default pose, action scaling, and joint limits. These shared contracts let the runtime detect incompatible exports rather than silently interpret an action incorrectly.
Evaluation progresses from Isaac Lab to MuJoCo, then to a closed loop with the actual C++ planner and the message format used by the A3 runtime. Physical racket, table, and net contacts are evaluated in MuJoCo before hardware use. The deployment stack includes a native C++ controller and ROS 2 integration.
A complementary model-based serve
The repository also includes a separate serving workflow. It searches for an outgoing ball trajectory, derives the racket’s impact velocity and orientation, then uses damped-least-squares inverse kinematics to convert a Cartesian racket path into arm motion.
The serving video in the gallery shows this MuJoCo application. Its high-level A3 runtime commands the arms while the vendor controller retains the waist, neck, and legs. It is distinct from the learned whole-body return policy.
Serving workflow & demoCurrent evidence and open questions
The published October 13, 2026, 30-ball MuJoCo run of model_14600 recorded 30 racket contacts and 30 measured legal landings, with no falls during the continuous motion interval. It included 10 forehand and 20 backhand shots.
The full validation gate did not pass. Station and support-foot behavior, recovery, final heading, event identity and timing, and forehand coverage still failed checks. These are simulator results for a specific run, not a hardware success rate or a completed qualification.
The current system is station-anchored, with in-place recovery. Free locomotion, spin-aware shot planning, opponent adaptation, and shot strategy remain outside the current scope.
Read the experiment reportHOPE, developed by Hitch Interactive with UC Berkeley ROAR. Cover illustration based on the repository’s A3 model.







