Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline

This research introduces Self-Verified Distillation, an algorithm enabling large language models (LLMs) to improve themselves using only unlabeled prompts. It involves generating, self-verifying through multi-stage checks, and then training on self-curated datasets, without external teachers.