Agentic Intelligence Lab

JoMoHa: How Should We Optimize Model Weights and the Harness Together?

Joint work by Xudong Wu, Guangchen Lan, Mingqi Yuan, and Jiayu Chen, from The University of Hong Kong and Salesforce.

TL;DR: A harness that works well for one model checkpoint can get in the way of the next. With JoMoHa, the harness writer and the task Solver learn from the same execution. They share the task reward, but each has a critic that estimates expected return in the context of the collaboration. JoMoHa reaches 97.0% on ALFWorld and averages 70.3% across three benchmarks.

Paper · Code · Original research blog

JoMoHa at a glance: why alternating updates can fail and how the two policies learn together.
JoMoHa at a glance: why alternating updates can fail and how the two policies learn together.

What goes wrong when we alternate?

The sponge was already clean. All the Solver had to do was put it on a cart. But the harness, the code that tracks progress and guides the model through the task, had missed the update and kept sending the Solver back to clean it. The run hit the 50-turn limit. Without that harness, the same Solver finished in 14 turns.

That was one failure in a long-phase alternating training run on ALFWorld. We trained the Solver with the harness fixed, then trained the harness writer with the Solver fixed. Success initially rose from 83.0% to 90.5%, then fell to 31.5%. When we evaluated earlier and later Solver checkpoints with the same learned harness, it interfered more with the later Solver. But the later Solver did worse on its own, too. The harness affects not just how a model executes a task, but also what it experiences during training.

What should the two policies learn from each other?

To our knowledge, JoMoHa is the first reinforcement-learning framework to jointly train a policy that writes executable harnesses and a policy that acts under them. In each paired task attempt, the Engineer generates a harness and the Solver uses it. Both learn from the same task reward, each updating from the tokens it generated.

The reward tells us what happened to the pair, but it does not set the same expectation for both roles. We give each policy its own critic. Each estimates expected return from the policy’s history and a short summary of the task, training version, and execution conditions. This extra context is for the critics, not the policies: it lets each learner compare the outcome with what was expected under the conditions in which the pair worked.

We also normalize returns separately for each role, since writing a program and acting over many turns can give returns on different scales. We weight updates using the partner’s recorded trajectory to account for how its policy has changed. And extra Solver updates let it adapt to sampled harnesses without another change to the Engineer’s weights.

JoMoHa s training loop: shared task reward, separate critics, and coordinated updates paper Figure 3 .
JoMoHa’s training loop: shared task reward, separate critics, and coordinated updates (paper Figure 3).

Does coordination improve the pair?

On ALFWorld, joint training without the coordination mechanisms reached 88.5%. Adding critic context alone brought it to 92.5%, the largest gain among the individual additions. The full method reached 97.0%.

The harness itself improved, too. With the later Solver’s weights held fixed, replacing the initial harness with the later one raised success from 92.5% to 96.5% in the cross-checkpoint evaluation. In the main comparison, JoMoHa led the evaluated methods on ALFWorld, WebShop, and DBBench, averaging 70.3%.

ALFWorld learning curves comparing JoMoHa with the frozen-Engineer, weight-only, and harness-only variants paper Figure 1 .
ALFWorld learning curves comparing JoMoHa with the frozen-Engineer, weight-only, and harness-only variants (paper Figure 1).

Paper figures reproduced from JoMoHa under CC BY 4.0.

Previous post
GPT-6-Astra Edits a VLA Policy Directly