Engineering and algorithmic interventions for multimodal post-training at Microsoft scale

This post describes five engineering and algorithmic interventions developed at Microsoft to stabilize reinforcement learning post-training of multimodal agents for Copilot at production scale.

• •Staged objective curriculum trains exclusively on verifiable signals (tool syntax, format compliance) for the first 30% of training, then linearly phases in preference signals to prevent premature specialization
• •Effective Sample Size (ESS) monitoring detects gradient estimator collapse ~35 epochs before learning stalls, triggering near-miss trajectory injection and adaptive KL penalty to restore outcome contrast
• •Variance-corrected normalization scales each trajectory's gradient contribution by per-source reward variance and sqrt(trajectory_length) to prevent long trajectories from dominating the gradient budget
• •Constraint shaping replaces hard penalty cliffs with shaped boundaries so agent orchestrators do not retreat to degenerate safe behaviors such as serial collapse or unnecessary su