CWI — Composite Humanoid Whole-Body Imitation System for Loco-Manipulation (RA-L 2026)

One Policy, Two Kinds of Motion Data

Accepted to IEEE Robotics and Automation Letters, June 2026.

Wenqi Ge*, Junde Guo*, Zhen Fu, Shunpeng Yang, Jiayu Chen, Hua Chen

LimX Dynamics · The University of Hong Kong · SUSTech · HKUST · Zhejiang University

_{*Equal contribution}

Project page · arXiv:2606.27676 · Video (YouTube) · Video (Bilibili)

CWI on a full-size LimX Oli: squatting to pick up toys and drop them in a basket (a-1), standing up and walking off carrying it (a-2), opening a door (b), hanging cups on a rack (c), striking a suspended drum (d), assembling a small part (e), twisting a bottle cap (f).

A humanoid that picks a toy off the floor and carries it across the room is solving two problems at once. The legs have to hold a commanded velocity and height while the arms swing the base around. The arms have to reach wherever the operator points, with motion that stays inside what a human body would actually produce. Push hard on one objective and the other tends to give.

Human motion capture is the obvious source of priors for both halves. That is also where the trouble starts.

Where full-body MoCap breaks down

AMASS and similar corpora are not evenly useful across the body. Upper-body trajectories are rich: people reach, gesture, and handle objects in every direction, which is close to what a manipulation policy needs. Lower-body trajectories are another matter.

AMASS lower-body statistics against our training command range (blue). Pelvis height piles up near standing, 0.85 to 0.90 m, while we command down to 0.17 m. Base velocities cluster at the origin, while we command up to 1.0 m/s forward and 1.2 rad/s in yaw.

Two things go wrong. The clips do not cover the corners of the command box we need for deployment, so a policy trained to track them never learns to hold 0.3 m/s at a 0.4 m squat. And many of the clips that do move are too aggressive for a robot that is carrying something, so they have to be filtered out.

Both standard responses to this cost something. Filtering the corpus for lower-body stability throws away the upper-body motion in those same clips, which was perfectly good. Skipping MoCap for the legs and shaping rewards or sampling joint commands instead leaves the upper body without human statistics, and the arm motion drifts away from anything a person would produce.

Decouple the data, keep the policy whole

Several existing loco-manipulation controllers decouple the architecture: one policy for the legs and another for the arms, or RL for the legs and PD control on top for the arms. CWI keeps a single whole-body policy over all 31 DoF and decouples only what each half learns from.

The upper body trains on the full AMASS corpus with no filtering. The lower body trains on roughly ten curated expert-quality clips per class, one class for omnidirectional walking and one for squatting.

The CWI framework. Large-scale upper-body data drives manipulation, compact curated clips drive locomotion style through two discriminators, a multi-critic PPO stage trains the unified policy, and distillation produces a student driven by velocity/height commands plus bimanual hand poses.

The reason the upper-body corpus needs no filtering is a change of frame. We express every upper-body reference relative to the robot’s base. Once root motion is subtracted, a clip of someone spinning or jumping still contains a clean, human-feasible arm trajectory. The instability lives in the root, and the root is exactly what we drop.

Two details in how the body is partitioned turn out to matter later. The waist (yaw, roll, pitch) goes into the upper-body group, so torso bending is supervised by the same AMASS clip as the arms rather than being treated as part of the locomotion problem. The neck and head come out of the reference entirely, since AMASS contains very few head motions; instead the head gets a 2-DoF command sampled within its mechanical range each episode and a separate quadratic tracking term, which lets a VR headset drive it directly at deployment.

The upper-body reference combines joint targets q_u^ref ∈ R^17 with position, orientation, and keypoint-based pose for the shoulder, elbow, wrist, and hand links. The lower-body reference for the discriminators carries base velocity and height, the 12 leg joint positions and velocities, and hip/knee/ankle link states.

Two discriminators, one per gait class

The lower body gets its supervision through Adversarial Motion Priors rather than frame-by-frame tracking. Each discriminator scores a four-frame window of lower-body state transitions, with upper-body signals excluded so the prior stays purely about the legs, and trains with a least-squares GAN objective plus a gradient penalty.

Which discriminator is active depends on the command: D_walk when the velocity command is nonzero, D_squat when it is zero and the policy is tracking height. Both feed a continuous style reward that pays off states the discriminator classifies as expert-like.

The important consequence is that the curated clips act as a style prior, not a tracking target. Locomotion commands are sampled uniformly across the entire deployment range during training, far outside anything the ten clips demonstrate. The discriminator answers “does this look like expert walking”, not “does this match frame 47 of clip 3”, so command coverage and motion quality stop competing for the same data.

Walking and squatting are distinct enough that pooling them into one discriminator averages the two modes and blurs both, which the ablations below confirm.

Three reward groups, three critics

The full reward splits into locomotion tracking, upper-body tracking, and style:

Standard regularizers (projected-gravity alignment, action smoothness, torque penalties) are assigned to whichever group they primarily affect.

A single critic has to explain all three signals with one value function, and the adversarial style reward is high-variance enough to drag that estimate around. We give each group its own value head, compute group-wise GAE advantages, normalize each within the minibatch, and combine them with weights before the standard PPO clipped surrogate.

The multi-critic variant reaches a higher final reward and converges faster on both velocity and hand-position error.

Distilling down to two hand poses

Single-stage RL for whole-body tracking tends to come out jerky, so we train a teacher with privileged state and the full upper-body reference, then distill.

The student sees only a history of proprioceptive observations plus a reduced command: velocity and height for the legs, and an 18-dimensional bimanual hand pose (a 9D keypoint feature per hand) for the arms. Distillation combines PPO on the teacher’s reward with an action-matching BC term, querying the teacher on-policy over the student’s own state distribution and annealing the imitation weight.

Reducing the command to hand poses is a deployment decision. A VR operator supplies end-effector targets, not joint angles, and reconstructing joint targets through online IK risks discontinuities near singularities. Distilling directly on hand poses sidesteps that and keeps the student pinned to the AMASS manifold. The cost is a small open-loop accuracy gap against the joint-command interface.

How it compares

We evaluate on a 31-DoF LimX Oli (1.65 m, 50 kg), trained in IsaacLab with commands sampled over v_x ∈ [−0.5, 1.0] m/s, ω_yaw ∈ [−1.2, 1.2] rad/s, h ∈ [0.17, 0.9] m, and the usual domain randomization over mass, friction, sensor noise, and latency. The AMASS upper-body corpus is split 85/15, and evaluation uses sequences the policy never saw.

Because the original works target different robots, we re-implemented every baseline on Oli (marked with *) and unified observation composition, reward terms and weights, and PD gains across all methods, so the comparison isolates framework design and data usage.

Lower is better on every row except success rate. Position and orientation errors aggregate over all upper-body keypoints.

FALCON* carries higher arm tracking error, consistent with its separate upper and lower policies. HOVER* loses robustness once upper-body motions are paired with base commands outside its training distribution. HOMIE* shows larger joint-level deviation, which fits a PD-controlled upper body asked to follow non-smooth references.

The one metric where CWI does not come first is success rate, where FALCON* leads by about a tenth of a percentage point. Our policy holds strict upper-body tracking even at the boundary of the command box, and that occasionally costs a little stability.

Which pieces are doing the work

The ablations use the deployment-interface student, so the manipulation columns report end-effector error over the two hands rather than the full-keypoint aggregate above. We add a relaxed DTW distance d_dtw against reference clips as a naturalness measure, and the 95th-percentile joint acceleration q̈_p95 as a smoothness proxy.

_{† The single-stage variant fails upper-body tracking outright, so its locomotion numbers are reported for transparency but not ranked.}

Removing distillation is the sharpest failure. A single-stage policy cannot learn manipulation from the sparse hand-pose command alone, and end-effector error goes from 42.91 mm to 173.2 mm. The teacher has to learn whole-body control with dense references first; the student inherits it.

Dropping AMP barely touches command tracking but collapses gait quality: d_dtw jumps from 0.452 to 1.413 rad, with frequent stance violations and foot slips. Pooling the two discriminators into one is the same failure in milder form, with d_dtw at 0.615 and joint acceleration up from 28.8 to 35.1 rad/s². End-effector accuracy stays near 42.9 mm in both cases, so the dual-discriminator design buys lower-body style without charging manipulation for it.

The result we find most telling is the AMASS upper-body ablation. Swapping human references for randomly sampled joint targets degrades manipulation as expected (42.91 to 62.32 mm, joint acceleration 28.8 to 48.2 rad/s²), but it also degrades every locomotion metric: velocity error 0.100 to 0.131 m/s, yaw rate 0.183 to 0.249 rad/s, height 19.65 to 24.20 mm. The architecture supplies the structural decoupling. Only the data supplies upper-body motion whose statistics the legs can absorb, and a jerky upper body is a disturbance the legs then have to reject.

On the robot

Deployment runs off a Meta Quest headset and its hand controllers, with no MoCap suit and no exoskeleton cockpit. The operator’s hand poses and a velocity/height command are the entire interface.

The tasks in the opening figure cover precision work (twisting a bottle cap, assembling a small part), whole-body coordination (opening a door, striking a suspended drum), and long-horizon sequences (picking up toys, placing them, then carrying the basket away). The policy holds stable locomotion throughout.

Four phases of a teleoperated box lift: approaching and squatting (A), grasping and starting the lift (B), raising with both hands (C), standing up with the box overhead (D). Below, left and right hand distance from the base against waist pitch over time.

Watch the waist pitch. It rises as the hands extend forward and down, then straightens as the robot comes upright with the box. The operator never sent a torso command; there is no torso command in the interface. The coupling between hand reach and torso posture is in the AMASS clips because that is how people reach, and putting the waist in the upper-body group meant one clip supervised both. The composite objective keeps that coupling intact while the legs stay stable underneath it.

Limits and what comes next

Behavior coverage is bounded by the motion libraries and, more tightly, by the command interface. Bimanual hand poses plus velocity and height cannot express pressing a pedal with a foot or bracing against a surface with an elbow.

That same compactness is useful going forward. A portable teleoperation rig that works in ordinary rooms is a data collection instrument, and a small, task-oriented command space is structured supervision for higher-level autonomous policies, vision-language-action models included. Collecting multimodal demonstrations in everyday scenes is where we are taking this next.

Citation

@article{ge2026cwi,
  author  = {Ge, Wenqi and Guo, Junde and Fu, Zhen and Yang, Shunpeng
             and Chen, Jiayu and Chen, Hua},
  title   = {{CWI}: Composite Humanoid Whole-Body Imitation System
             for Loco-Manipulation},
  journal = {IEEE Robotics and Automation Letters},
  year    = {2026},
  doi     = {10.1109/LRA.2026.3710366}
}