Calibrating LLM classification confidences
LLMs are often used to classify text content, achieving strong zero-shot accuracy. However, the classification confidences tend to be unreliable. Incorrect classifications are often assigned high confidence, and vice versa. In this post we look at two methods for inferring classification confidences from LLMs, and show how to (mostly) fix the confidence estimates using post-hoc calibration.
