Score Centering and Importance Sampling Combined to Fix LLM RL Training-Inference Mismatch

Score centering and importance sampling are complementary methods for addressing training-inference mismatch in LLM reinforcement learning, according to a technical analysis. Score centering removes gradient drift by subtracting the expected sampler score from each token's score, while importance sampling reweights token-level contributions toward the trainer distribution. Combining both methods corrects distribution mismatch and eliminates gradient drift without unbounded importance ratios, with empirical results showing the combination outperforms score centering alone under severe mismatch such as stale rollouts.
Event version 1.
Key insights
Score centering eliminates gradient drift
Under training-inference mismatch, tokens are sampled from a sampler policy while gradients are computed using a trainer policy, producing a spurious gradient component called gradient drift. Score centering subtracts the expected score under the sampler from each sampled token's score, removing drift while preserving the reward-dependent covariance between rewards and token-level scores.
Importance sampling corrects distribution mismatch but risks instability
Importance sampling reweights gradient contributions using ratios of trainer and sampler probabilities at the token level. Large importance ratios can cause high-variance gradients and unstable updates, and clipping or masking extreme ratios reintroduces the gradient drift that score centering aims to eliminate.
Combining both methods is straightforward
The combination computes importance-weighted token-level scores and subtracts the expected importance-weighted score under the sampler at each prefix, rather than centering the original scores. This addresses both distribution mismatch and gradient drift without relying on unbounded importance ratios.
Empirical performance depends on mismatch severity
Score centering alone performs well when training-inference mismatch arises from small numerical discrepancies such as mild quantization, with little additional benefit from importance sampling. Under severe mismatch such as stale rollouts, combining score centering with truncated or masked importance sampling achieves better performance than score centering alone.