Psychometric validity for LLMs

An instrument validated on humans carries no guarantee about machines. Human validation rests on assumptions — stable traits, consistent self-models, item interpretations shared across a population — that may simply not hold for a stochastic text generator. Administering a personality inventory to a model and reporting the score as if it were a trait measurement skips the entire evidentiary chain psychometrics exists to provide.

The core checks translate directly. Internal consistency (Cronbach's alpha): do items that supposedly measure one construct pattern together in the model's responses? Split-half reliability: do two halves of the instrument agree? Alternate-forms reliability: does a paraphrased version of the instrument yield the same scores — or was the original measuring memorized surface forms? Test-retest logic becomes replication: is the score stable across repeated administrations at the same settings?

Structured checklists for LLM psychometrics (e.g. Löhn et al., 2024) organize these criteria so a study can document, item by item, which validity evidence it has and which it lacks. Publishing the checklist alongside results lets readers judge how much weight the scores can bear.

AIScholar builds in Cronbach's alpha, split-half (Spearman-Brown) and alternate-forms reliability analyses, plus a psychometric validity checklist and contamination probes, so instrument-based studies carry their validity evidence with them.

• Walkthrough: psychometric assessment of an LLM
• How psychologists use AIScholar

• In-context-learning contaminationDetecting when instrument scores reflect memorization or in-prompt adaptation instead of the construct.
• LLM-as-a-JudgeUsing a judge model to code free-text responses against a rubric, with human calibration.
• Machine psychologyStudying LLMs as behavioral subjects with experimental methods from psychology.

Browse all 11 glossary terms