Semantic uncertainty

Because a language model's output is sampled from a probability distribution, asking the same question repeatedly yields varying answers. Raw text differences overstate this variation — "Yes, that's correct" and "That is right" are different strings but the same answer. Semantic uncertainty fixes this by first grouping responses into meaning clusters, then measuring the spread of the distribution over clusters. Entropy is zero when every response means the same thing and rises as answers scatter across distinct meanings.

High semantic uncertainty in an experimental condition is diagnostic. It can indicate that the prompt is ambiguous, that the model genuinely has no stable disposition on the question, or that temperature is set too high for the measurement you want. Either way, a condition where the model gives ten semantically different answers should not be summarized by a single "typical" response — the diversity itself is the finding.

Consistency is therefore a first-class outcome variable, not just noise. Studies can compare semantic entropy across conditions, models, or parameter settings — for example, testing whether higher temperature increases the semantic diversity of answers to open-ended questions, or which of several models answers a class of questions most consistently.

AIScholar computes semantic entropy per condition by clustering responses by meaning, flags high-diversity conditions, and lets you use entropy as a dependent variable in analyses alongside labels, numbers, and judge-rated scores.

• Analysis & visualization features

• Variance decompositionPartitioning outcome variance into condition, model, prompt-variant, and residual components.
• Prompt sensitivityMeasuring how results change under semantically-equivalent rewordings of a prompt.
• LLM-as-a-JudgeUsing a judge model to code free-text responses against a rubric, with human calibration.

Browse all 11 glossary terms