Direct¶
Overview¶
The MLLM controls the robot one motion at a time. Each turn it calls a motion tool, usually move with an
absolute grasp-site pose for one or both arms. Inverse kinematics turns the pose into joint targets, and the
model sees the resulting images and state before its next call.
Based on uni-robot-agent's Clean mode, a four-tool baseline, and on direct LLM control as evaluated in GPT-6 Astra on RoboDojo.
| Variant | Description |
|---|---|
direct_robodojo.py |
RoboDojo, two ARX X5 arms |
direct_jev_robodojo.py |
Keeps the whole history and compacts it with Jev, as code_jev does |
Usage¶
Tools¶
| Tool | What it does |
|---|---|
move(targets, profile, max_steps) |
Moves one or both arms toward absolute grasp-site poses [x, y, z, roll, pitch, yaw], with an optional gripper opening per arm |
hold(steps) |
Advances physics for 1 to 40 control steps while the arms hold |
capture_scene() |
Returns fresh images and state without moving |
finish(status, summary) |
Ends the episode when the model judges the task done |
Implementation details¶
- Control point. Poses refer to each arm's grasp site, the centre between the finger pads, 14.93 cm in
front of link6. The offset comes from uni-robot-agent's
x5_grasp_site.urdf. Orientations are fixed-axis XYZ Euler angles. The observation adds thisposeto each arm. - Profiles.
standardmoves 4 cm per control step and stops within 3 cm and 0.1 rad;approach4 cm, 2 cm and 0.05 rad;precision1 cm, 5 mm and 0.05 rad. Rotation uses part of each step's displacement, because rotating link6 also moves the grasp site. - Stopping.
movestops when every pose is within tolerance and requested fingers have stopped moving (targets_reached), when an arm improves by less than a tenth of the tolerance over three steps (stalled), or aftermax_steps(at most 24). Feedback gives each arm's position and orientation error. - Inverse kinematics. RoboDojo's native solver targets link6. A waypoint whose solution would jump a joint
by more than 1 rad is rejected, and
movereturns the error as feedback. - Gripper. Openings are normalized: 0 closed, 1 open. The observation reports both the measured opening and the last command, because the fingers can stop on an object before reaching the command.
Functional checks¶
Note
One episode per task on RoboDojo seed 0, with GPT-6 Astra through an OpenAI-compatible API, run on 2026-11-17. These check that the agent works; they are not success rates.
| Task | Score | Decisions | Tokens |
|---|---|---|---|
| general_pickup | 100 (success) | 6 | 35k |
| stack_blocks | 100 (success) | 46 | 265k |
| stack_bowls | 15 | 60 (budget) | 339k |
| push_T | 0 | 60 (budget) | 363k |
In stack_bowls, 11 of the 58 move calls failed on inverse kinematics and 5 stalled.
direct_jev gave no clear gain (2026-11-19, one episode each): stack_blocks_by_language still succeeded, with
329k tokens instead of 207k, and put_bottles_into_dustbin scored 25 instead of 10 but still ran out of steps.
Direct's failures come from inverse kinematics and step limits more than from its context.