Continual Policy Distillation from Distributed Reinforcement Learning Teachers
Abstract
Continual Reinforcement Learning (CRL) aims to develop lifelong learning agents to continuously acquire knowledge across diverse tasks while mitigating catastrophic forgetting. This requires efficiently managing the stability-plasticity dilemma and leveraging prior experience to rapidly generalize to novel tasks. While various enhancement strategies for both aspects have been proposed, achieving scalable performance by directly applying RL to sequential task streams remains challenging. In this paper, we propose a novel teacher-student framework that decouples CRL into two independent processes: training single-task teacher models through distributed RL and continually distilling them into a central generalist model. This design is motivated by the observation that RL excels at solving single tasks, while policy distillation -- a relatively stable supervised learning process -- is well aligned with large foundation models and multi-task learning. Moreover, a mixture-of-experts (MoE) architecture and a replay-based approach are employed to enhance the plasticity and stability of the continual policy distillation process. Extensive experiments on the Meta-World benchmark demonstrate that our framework enables efficient continual RL, recovering over 85% of teacher performance while constraining task-wise forgetting to within 10%.
Method Overview
Distributed PPO teachers
Each task is first solved by a specialized PPO teacher. The resulting demonstrations provide stable state-action supervision for the central policy.
Imitation learning distillation
The student learns teacher behavior through policy distillation, turning sequential RL acquisition into a supervised continual imitation learning problem.
Expandable Transformer-MoE
Sparse MoE layers give the student scalable capacity, while task embeddings condition the policy on latent context inferred from trajectories.
During continual updates, the framework combines expert expansion, diversity-aware trajectory replay, and hierarchical parameter masking. Shared Transformer components are frozen after the initial stage to reduce representation drift, while newly added experts and task embeddings remain plastic enough to absorb incoming skills.
Experimental Highlights
| Setting | Main result | Interpretation |
|---|---|---|
| MT10 multi-task distillation | 88.9% success rate | Matches 98% of the PPO teacher average of 90.8%. |
| Two-phase MT25 continual distillation | >85% teacher performance, <10% forgetting | Shows the replay and masking strategy preserves prior tasks during phase 2. |
| Five-phase MT25 continual learning | 69.4% final accuracy, 2.3% BWT | Maintains positive backward transfer in the harder five-stage stream. |
The ablations isolate the contribution of each component: removing task embeddings drops final accuracy to 57.9%, replay-only obtains 57.2% with -11.0% BWT, and expert-only expansion collapses to 28.0% with -51.3% BWT. The full method works because expansion provides capacity, replay anchors old behavior, masking limits destructive updates, and task embeddings give the policy usable task context.
Takeaways
Why distillation?
PPO remains effective for specialized single-task teachers, while imitation learning gives the central model a more stable training signal than direct multi-task online RL.
Why MoE expansion?
Expert expansion increases capacity over time without forcing every new task to overwrite the same dense parameters.
Why replay and masking?
Replay keeps representative prior trajectories in the training mix, and masking restricts parameter updates that would otherwise erase old skills.
BibTeX
@misc{he2026continualpolicydistillation,
title={Continual Policy Distillation from Distributed Reinforcement Learning Teachers},
author={Qijun He and Yuxuan Li and Mingqi Yuan and Xiaoquan Sun and Wentse Chen and Jeff Schneider and Jiayu Chen},
year={2026},
eprint={2601.22475},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2601.22475}
}