Can you give a language model a validated scale?
A walkthrough of a psychometric study: administering a risk-attitude instrument to models item by item, checking internal consistency and alternate-forms reliability, probing for training-data contamination, and working through a validity checklist before believing a single score.
Stage 1 · Hypothesis
Measurement first, claims second
It is easy to feed a model a questionnaire and report a score. It is much harder to show the score means anything. This study inverts the usual order: the primary hypotheses are about measurement properties, and only the secondary one is substantive.
| ID | Prediction |
|---|---|
| H1 (reliability) | Item responses to a 10-item risk-attitude scale will show acceptable internal consistency (Cronbach's α ≥ .70) within each model. |
| H2 (alternate forms) | Scores on the original items and on paraphrased alternate-form items will correlate strongly, indicating the construct — not the exact wording — drives responses. |
| H3 (substantive, conditional) | If H1 and H2 hold, models will differ systematically in mean risk-attitude scores. |
Stage 2 · Design
Items as a factor, one item per trial
Because every AIScholar trial is a single prompt and a single response, the instrument is administered one item per trial — which is also the methodologically cleaner choice, since it prevents earlier items from contaminating later answers within a context window.
| Design element | Choice |
|---|---|
| Factors | item (10 levels) × form (original, alternate) × model (3 OpenRouter models) |
| Crossing strategy | Full factorial |
| Replications | 25 per cell — each model answers each item 25 times, giving a response distribution per item |
| Expected output | Numeric 1–7, range-validated at extraction |
| Trial count | 10 items × 2 forms × 3 models × 25 replications = 1,500 trials |
Treating replications as the unit of analysis is the key departure from human testing: a person answers an item once; a model's "answer" is a distribution, and the study analyzes its central tendency and spread.
Stage 3 · Prompts
Neutral administration, versioned items
Each item is a snippet; the alternate form binds paraphrased versions of the same items. The administration wrapper is deliberately minimal to avoid persona effects:
Please respond to the following statement on a scale
from 1 (strongly disagree) to 7 (strongly agree).
Reply with a single number only.
Statement: {{scale_item}}| Snippet · version | Bound to | Text |
|---|---|---|
| scale_item_3 · v0 | form = original | "I would take a job with variable pay over one with a fixed salary." |
| scale_item_3 · v1 | form = alternate | "Given the choice, I'd pick work where my earnings fluctuate rather than a set wage." |
Stage 4–5 · Execution & Collection
Range-validated numbers, flagged refusals
The numeric output declaration (valid range 1–7) means extraction, range checks, and the rescue parse are automatic. Responses like "As an AI, I don't have preferences…" are classified as refusals rather than silently dropped — and the per-condition refusal rate becomes reportable data. A pilot run (one trial per cell) verifies formats before the 1,500-trial spend.
Stage 6 · Coding
Contamination probes before you trust the scores
A published scale is almost certainly in the training data. The study therefore runs AIScholar's contamination toolkit alongside the main design:
| Check | What it does |
|---|---|
| Memorization probe | Prompts each model with truncated scale items to test whether it completes them verbatim — a direct signal the instrument was memorized. |
| Modified-item generation | Generates content-matched but surface-novel items for any flagged instrument, so the construct can be measured with wording the model has not seen. |
| ICL detection | Screens responses for signs the model recognized the instrument and is role-playing "how one answers this famous scale". |
Stage 7 · Analysis
Reliability, then the validity checklist
The analysis stage computes Cronbach's α and split-half (Spearman-Brown) reliability per model, alternate-forms correlations across the two item sets, and only then — if measurement holds — the substantive model comparison.
| Model | Cronbach's α | Split-half | Alternate-forms r |
|---|---|---|---|
| Model A | .84 | .81 | .77 |
| Model B | .71 | .68 | .52 |
| Model C | .43 | .39 | .21 |
These numbers are illustrative
An illustration like Model C's row is the important one: reliability that low means that model's "risk attitude" is not a coherent measured construct, and its mean score should not be interpreted — a conclusion most casual LLM-questionnaire papers never check for. These numbers show the output shape, not findings from a real run.
The Psychometric Validity Checklist (after Löhn et al., 2024) is filled in from these results, item by item, and travels with the project into the write-up.
Stage 8 · Write-Up
Reporting scores you can defend
The drafted Method section documents the one-item-per-trial administration, sampling parameters, and contamination checks — exactly the details reviewers of machine-psychology papers now ask for. The reproducibility package carries the full item text (original, alternate, and any generated modified items) with the trial-level data.
Stage ✓ · Honest limits
What this design can and cannot claim
Even with strong reliability, a model's scale score is a property of its response distribution under this administration procedure — not evidence of an inner disposition. Construct validity against external criteria (does the "risk score" predict risky choices in other tasks?) needs a separate study — the persona × task factorial is a natural companion for that. Sampling temperature also shapes score spread, so report it as part of the measurement procedure, not a nuisance detail.
- Cronbach's α
- Alternate forms
- Contamination checks
More example studies
- Persona × task factorial studyA 3×3 full factorial built with the snippet library and variation grid, analyzed with factorial ANOVA.
- Coding open-ended responsesLLM-as-a-judge coding with a versioned scheme, multi-judge agreement statistics, and human gold-standard review.
- Analyzing an existing datasetA bring-your-own-data study: upload existing responses, code them with a judge, and analyze without re-running trials.
Run this study yourself.
Every walkthrough on this page maps to real platform features. Start a project and follow along.