Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs

Kirill Skobelev1, Eric Fithian2, X.Y. Han2

NeurIPS 2026 The Fortieth Annual Conference on Neural Information Processing Systems Workshop Transitioning from Pre-Training to Post-Training

Abstract

Recent work by Doshi and Hauser (2024), Bisbee et al. (2024), and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend to be under-diverse: they repeat or resemble one another more often than responses from the population they are meant to represent, a phenomenon known as mode collapse. In this work, we show that whether mode-collapse—or its opposite—occurs depends on the specific model and dataset used. Further, with sufficient supervised fine-tuning (SFT) data, LLM output diversity converges toward that of the target distribution from which fine-tuning data are sampled. To quantify this comparison, we measure the probability that two responses sampled independently conditional on the same fixed prompt coincide (collide). In one set of experiments, we use exact token sequences; in the other, we use the generalised version of that—the expected similarity between responses under a kernel. We define miscalibration as a nonzero model-minus-target collision gap. We derive a bias–variance decomposition of the expected gap between the model's and target's collision probabilities, showing that SFT is not inherently biased toward mode collapse or its opposite: finite-sample SFT can leave a model either under- or over-dispersed, depending on the model and dataset. Finally, we show that the absolute gap is bounded by the square root of the Kullback–Leibler (KL) divergence from the target distribution to the model. Consequently, a model sufficiently close to optimal under population cross-entropy cannot exhibit arbitrarily miscalibrated diversity. We test the decomposition and the bound in three experiments: (1) we fit 100 small transformers at each of 100 log-spaced sample sizes on each of two synthetic order-16 languages; (2) we fine-tune four LLMs using low-rank adaptation on responses from the General Social Survey, American National Election Studies, and World Values Survey; and (3) we repeat Experiment (2) on CodeNet, a dataset of human solutions to coding tasks, measuring program similarity with normalized Zhang–Shasha edit distance between canonical abstract syntax trees. We find substantial heterogeneity in over- and under-diversity across models and datasets. More target data moves model diversity toward the human (or synthetic target) level in all experiments, consistent with our theoretical predictions. These results show that diversity miscalibration can arise from finite-sample error and shrink as SFT better approximates the target distribution. Accordingly, as the sample size of target-distribution data increases, model diversity moves toward the target level.

Main results

Figure 1. LLM diversity depends on the model and target distribution, and fine-tuning can improve its calibration. (a.i) Independent outputs from the same LLM and prompt can collide or differ. (a.ii) Notation for collision probability. (a.iii) The collision ratio R: values above 1 indicate mode collapse, and values below 1 indicate over-dispersion. (b) Decomposition of the expected collision ratio in the two settings of Experiment 1. The target–bias alignment term can offset the nonnegative variance and squared-bias terms, allowing either sign of miscalibration. Here bh = E[qh] − ph is the bias at context h, with E over independent fits. The waterfall bars start at R = 1 and show context-averaged contributions normalized by C(ph), ending at E[R]. (c) Median collision ratio R for early-stopped survey and final CodeNet adapters against fine-tuning examples from Experiments 2–3. More target data moves diversity toward calibration. (d) Theorem 1 bounds the absolute model–target collision gap by the square root of their Kullback–Leibler divergence. Approaching optimal population cross-entropy therefore forces diversity calibration.
Figure 2. Experiment 1: decomposition of the normalized gap between model and target collision probabilities. The two synthetic languages have known target distributions, so repeated fits let us estimate the contributions of variance, squared bias, and target–bias alignment to E[R] − 1. Contributions are averaged over contexts and plotted against sampled sequences N. (a) shows pre-training from the diffuse prior (6.135M-parameter GPT, N from 32k to 1M, 2000 common contexts); (b) shows fine-tuning from the mode-aligned prior (27.97M-parameter GPT, N from 500 to 106). The black line shows the estimated E[R] − 1, averaged over contexts: positive values indicate mode collapse and negative values over-dispersion. The three contributions sum to this value. The two panels carry independent vertical and horizontal scales.
Figure 3. Experiment 2: sociological surveys (GSS, WVS, and ANES). Collision ratio RU = C(q)/ĈU (1 = calibrated) against fine-tuning examples (20 sample sizes for the GSS, 18 each for the WVS and ANES). ĈU estimates how often two distinct human respondents give the same answer; C(q) is the probability that two independent model answers match, using probabilities normalized over the listed options. Each panel shows the median collision ratio across evaluated question–group pairs for early-stopped adapters (blue line), with shaded bands spanning the 25th–75th and 5th–95th percentiles across pairs. The marker at n = 0 is the zero-shot base model, with vertical whiskers repeating the two bands; the horizontal axis is symmetric-log so that n = 0 is shown on the same axis as the sweep, and each panel has its own y-axis limits.
Figure 4. Experiment 3: CodeNet diversity under normalized Zhang–Shasha similarity. Collision ratio CZS(q)/CZS(p) against SFT examples (1 = calibrated). Lines show median finite collision ratios across problems; shaded bands span the 25th–75th and 5th–95th percentiles. At the full fine-tuning dataset, each model's median uses all 2,647 candidate problems, while base markers cover 2,646, 2,569, 986, and 2,646 problems in panel order. Panels have separate y-axes and broken horizontal axes.