LLM-as-a-Judge

Open-ended responses cannot be analyzed statistically until they are turned into structured data. Traditionally that meant human coders reading every response — accurate but slow and expensive at the scale LLM experiments produce (thousands of responses per study). LLM-as-a-Judge automates the reading: the judge model receives the response plus an explicit rubric and returns a coded value for each variable in the scheme.

The method's known failure modes are why validation matters. Judges can drift from the rubric's intent, exhibit leniency or severity biases, be sensitive to the exact wording of the rubric, and inherit blind spots from their own training. Defensible use therefore requires anchored rubrics with concrete examples, calibration against a human-coded gold standard, agreement statistics (Cohen's kappa, Krippendorff's alpha), and — ideally — multiple judges whose disagreements are surfaced rather than averaged away.

Judge-prompt sensitivity is a measurable property: rephrasing the rubric and re-coding a sample reveals how much the results depend on wording rather than on the responses themselves. If small rubric changes flip codes, the coding scheme needs tightening before its output can support conclusions.

AIScholar's coding stage supports single- and multi-judge coding with any OpenRouter model, human review and gold-standard overrides that become authoritative, versioned coding schemes, inter-rater agreement statistics, and a judge-prompt sensitivity check that quantifies rubric-wording dependence.

• Response coding features
• Walkthrough: coding 1,000 open-ended responses

• Psychometric validity for LLMsRe-establishing reliability and validity when human instruments are administered to models.
• Semantic uncertaintyShannon entropy over meaning-clustered responses — response consistency as a measurable outcome.
• Prompt sensitivityMeasuring how results change under semantically-equivalent rewordings of a prompt.

Browse all 11 glossary terms