Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs

This research addresses the lack of diversity in LLM outputs, attributing it to how models allocate probability mass across valid and invalid continuations during decoding. It introduces a validity-diversity framework that decomposes the problem into two complementary forms of miscalibration: order calibration and shape calibration.