Differences in Text Generated by Diffusion and Autoregressive Language Models

This research explores the intrinsic differences in text generated by Diffusion Language Models (DLMs) and Autoregressive Language Models (ARMs), finding that DLMs show lower n-gram entropy but higher semantic coherence and diversity. Controlled experiments reveal that DLM training objectives enhance coherence and diversity through bidirectional context, while decoding algorithms are responsible for entropy reduction.