JoMoHa Framework Jointly Trains Harness and Task Policies via RL

Researchers introduced JoMoHa, a reinforcement learning framework that jointly trains an executable harness generation policy and a task execution policy from the same task outcomes. JoMoHa outperformed weight-only and harness-only baselines on ALFWorld, WebShop, and DBBench, averaging 70.3% success and reaching 97.0% on ALFWorld. Separate token histories and value estimates for each policy prevented the incompatibility seen in naive alternating training.
Event version 1.
Key insights
Alternating training causes incompatibility
Naive alternating updates to model and harness cause the two components to drift apart. Performance first rises and then falls as the harness and model become incompatible, interfering with task execution and potentially undermining model training.
Two policies trained from shared outcomes
JoMoHa trains one policy to write an executable harness and another to act under it, both from the same task result. Each policy keeps separate token histories and value estimates, so the harness writer is learned inside the system rather than being a fixed external frontier model.
Highest success across three benchmarks
JoMoHa beats weight-only and harness-only baselines and achieves the highest success rates among evaluated methods on ALFWorld, WebShop, and DBBench. It averages 70.3% and reaches 97.0% on ALFWorld.
Ablations confirm learned harness writer
At fixed solver weights, learned harnesses improve on the initial code. Training the harness writer beats freezing it, and coordinating the two learners beats plain joint updates.
Recursive self-improvement from own trajectories
JoMoHa learns both task execution and runtime improvement from its own generated trajectories, performing recursive self-improvement (RSI).