Continual Distillation of Teachers from Different Domains

This research introduces Continual Distillation (CD), a new paradigm where a student model sequentially learns from a stream of teacher models without retaining prior access. It addresses challenges like unseen knowledge transfer (UKT) and forgetting (UKF) through Self External Data Distillation (SE2D), which uses external unlabeled data to stabilize learning across heterogeneous teachers.