Charlotte Peale: Uncertainty Quantification Beyond Calibration (February 5, 2026)

Calibration has emerged as a standard approach to uncertainty quantification, providing valuable insights into model reliability. However, for modern machine learning, calibration exhibits two fundamental limitations that restrict its utility:

1. Inability to decompose uncertainty: Calibration fails to distinguish between epistemic uncertainty (model-based) and aleatoric uncertainty (data-based). This distinction is vital for understanding prediction errors, especially in complex, subjective tasks (e.g., language modeling), and for determining whether collecting more data could improve performance.
2. Suboptimal error prediction: The uncertainty estimates produced by a calibrated model can be substantially worse than those derived from externally trained models specifically designed to predict a model’s error. This gap suggests that stronger notions of uncertainty quantification performance are required to guarantee a model’s ability to accurately self-assess its limitations.

This talk will overview two research contributions that address these fundamental limitations. First, we introduce higher-order calibration, a rigorous theoretical foundation for decomposing a model’s total uncertainty into its aleatoric and epistemic components, with formal guarantees relating the decomposition to real-world data distributions. We demonstrate the practical utility of this decomposition in uncertainty-aware model routing, where estimates are used to efficiently route queries to small models, larger models, or human experts. Second, we establish an equivalence between a model’s level of multicalibration and its competitiveness with externally-trained loss predictors. This equivalence reveals the precise conditions under which models can—or cannot—accurately assess their own limitations.

Видео Charlotte Peale: Uncertainty Quantification Beyond Calibration (February 5, 2026) канала Simons Foundation