Diagnosing instability in production-scale agent reinforcement learning

This post identifies a late-phase instability mechanism in production-scale reinforcement learning for tool-using agents, caused by tool-conditioned variance amplification.

• •Variance amplifies in post-tool contexts because tool calls expose the policy to low-support regions of the reference policy, inflating importance-weighted gradient estimates
• •Aggregate metrics (loss, reward, entropy, global KL) remain stable while tail growth occurs in tool-conditioned slices, causing misattribution to optimizer or global variance issues
• •The failure is diagnosed via 95th percentile of per-token log-ratios and effective sample size (ESS), sliced by pre-tool vs post-tool interaction mode
• •Constraining tool outputs (schema-constrained, distributionally narrow) substantially suppresses tail growth