on-policy distillation

Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

This paper introduces the On-Policy Diffusion Language Model (OPDLM) for transforming autoregressive models (ARLMs) into diffusion language models (DLMs). It addresses issues like knowledge loss and train-inference mismatch by employing On-Policy Distillation (OPD).