Keep the Tokens Flowing: Lessons from 16 Open-Source RL Libraries

This post surveys 16 open-source reinforcement learning libraries to understand how they implement asynchronous training architectures that decouple inference from training.

• •Synchronous RL training leaves GPUs idle because a single 32K-token rollout batch on a 32B model can take hours while training GPUs wait
• •The common solution is disaggregating inference and training onto separate GPU pools connected by a rollout buffer with asynchronous weight transfers
• •Seven comparison axes are used: orchestration primitives, buffer design, weight sync protocols, staleness management, partial rollout handling, LoRA support, and distributed training backends
• •Ray dominates orchestration (8 of 16 libraries), NCCL broadcast is the default weight transfer method, and staleness management ranges from dropping old samples to importance-sampling correction