XNews

Score Centering Technique Proposed to Stabilize LLM Reinforcement Learning

Score Centering Technique Proposed to Stabilize LLM Reinforcement Learning

A deep dive published on score centering explains why LLM reinforcement learning becomes unstable under training-inference mismatch and proposes an additive correction to the policy gradient. The technique subtracts the sampler's expected token score from each sampled token's score, and can be approximated using only the sampler's top-k token probabilities. The authors show best performance is often achieved by combining score centering with truncated or masked importance sampling.

Event version 1.

Key insights

Training-inference mismatch causes instability

Modern RL infrastructure separates rollout generation and policy updates into sampler and trainer engines. Numerical precision, quantization, different GPU kernels, and asynchronous training with stale rollouts cause token probabilities to differ between the two engines, introducing systematic biases into policy-gradient updates.

Gradient drift creates spurious learning signal

Under training-inference mismatch, a constant-reward setup produces a non-zero expected policy gradient, a spurious learning signal called gradient drift. This drift pushes the trainer toward the sampler regardless of the reward signal.

Feedback loop accumulates errors

Online RL refreshes the sampler from the trainer, creating a feedback loop. Biased updates to the trainer propagate to the sampler, and errors accumulate as the cycle repeats.

Score centering corrects drift additively

Score centering subtracts the expected score under the sampler from the score of each sampled token. The centered score has zero expectation at every prefix, removing the drift.

Efficient approximation and importance sampling combination

The correction can be approximated using only the sampler's top-k token probabilities, with k equal to 32 or 128, and the remaining probability mass approximated by the trainer's distribution. Best performance is often achieved by combining score centering with truncated or masked importance sampling, which helps with extreme mismatch such as very stale rollouts.

Source posts