Reading Calibrated Uncertainty from Language Model Trajectories

This research paper proposes a new method to quantify uncertainty in language models by tracing the cumulative path of per-layer MLP updates. By extracting eleven scale-invariant geometric features, a sparse linear probe is shown to outperform maximum softmax probability in evaluating uncertainty, especially with baseline miscalibration.