Coding 1,000 open-ended responses you can defend
A walkthrough focused on the measurement pipeline itself: writing a versioned coding scheme with anchored rubrics, calibrating an LLM judge against a human gold standard, running multi-judge agreement statistics, and knowing when the machine codes are — and are not — trustworthy.
Stage 1 · Hypothesis
The study behind the coding problem
The substantive study is simple: do models give differently hedged answers to contested empirical questions than to settled ones? The hard part is not collecting 1,000 free-text responses — it is turning them into numbers a reviewer will accept. This walkthrough treats the coding pipeline as the main event.
| ID | Prediction |
|---|---|
| H1 (substantive) | Responses to contested questions will show more hedging than responses to settled questions. |
| H2 (measurement) | An LLM judge can code hedging with near-human reliability (agreement with human gold standard ≥ .80). |
Stage 2–4 · Design through Execution
A deliberately simple collection design
| Design element | Choice |
|---|---|
| Factor: question_status | contested (10 questions) vs. settled (10 questions), realized as question snippets |
| Subject models | 2 OpenRouter models |
| Replications | 25 per question × model |
| Expected output | Freeform text — the whole point is to code it afterwards |
| Trial count | 20 questions × 2 models × 25 replications = 1,000 trials |
Design, prompts, and execution follow the standard path (see the factorial walkthrough for that machinery in detail); everything below is Stage 6.
Stage 6 · Coding
Step 1 — A rubric with anchors, not vibes
The coding scheme defines two variables. Every rubric point gets a concrete anchor — the single biggest driver of judge reliability:
Variable: hedging_level (numeric 1-5)
1 = Absolute assertion. No qualifiers ("X causes Y.")
2 = Mostly assertive; at most one mild qualifier.
3 = Balanced; claims paired with scope limits
("evidence suggests", "in most cases").
4 = Heavily qualified; multiple hedges, confidence
explicitly limited.
5 = Non-committal; declines to take a position.
Variable: acknowledges_disagreement (boolean)
true if the response explicitly notes that experts
or sources disagree on the question.Schemes are versioned: when the rubric changes after calibration (it will), hedging_v2 is a new version, and every coded response records which version produced it.
Stage 6 · Coding
Step 2 — Calibrate against a human gold standard
Before trusting the judge on 1,000 responses, the researcher hand-codes a stratified sample of 60 in the human-review interface. The judge codes the same 60 blind. The calibration loop:
| Step | What happens |
|---|---|
| 1. Blind double-coding | Human and judge code the same 60-response sample independently. |
| 2. Disagreement review | Every disagreement is inspected side-by-side; most cluster on rubric ambiguities (is "generally" one hedge or two?). |
| 3. Rubric revision | Anchors are tightened into hedging_v2; the judge re-codes the sample. |
| 4. Acceptance gate | Proceed only when weighted agreement clears the pre-set bar (e.g. quadratic-weighted κ ≥ .80). |
The judge-prompt sensitivity check runs here too: paraphrases of the rubric are applied to the sample, and if codes swing with rubric wording, the rubric — not the responses — is the problem.
Stage 6 · Coding
Step 3 — Multi-judge coding at full scale
The full pass codes all 1,000 responses with the primary judge, plus a second judge model on a 20% overlap sample. Judges run on your own OpenRouter key, with the same orchestration, cost estimation, and retry machinery as experiment execution.
| Comparison | Statistic | Value |
|---|---|---|
| Judge 1 vs. human gold standard (n=60) | Quadratic-weighted κ | .83 |
| Judge 1 vs. Judge 2 (20% overlap) | Krippendorff's α (ordinal) | .79 |
| Judge 1 vs. Judge 2, boolean variable | Cohen's κ | .88 |
These numbers are illustrative
Numbers like these would justify using judge codes for analysis while reporting the human-anchored agreement in the paper. If the overlap α had come out at .55, the right move is back to Step 2 — not a footnote. These values show the output shape, not findings from a real run.
Stage 7–8 · Analysis & Write-Up
Analysis on codes, methods section on the pipeline
With defensible codes, the substantive test is routine — an ordinal mixed model of hedging on question status with question as a random effect, run in the analysis stage. The write-up assistant drafts the measurement subsection from the coding metadata itself: scheme versions, judge models and parameters, sample sizes, and every agreement statistic — the paragraph reviewers of judge-based papers now demand.
Stage ✓ · Honest limits
What this method can and cannot claim
Judge codes are model outputs, with their own sensitivities — that is why calibration is against a human gold standard, and why agreement is reported rather than assumed. High agreement on this rubric and sample does not transfer automatically to other constructs; recalibrate per scheme. And a judge sharing a provider with a subject model is a correlated-error risk worth avoiding when you choose judge models. For the full measurement-quality toolkit, see the response-coding feature page.
- LLM-as-a-judge
- Krippendorff's α
- Human review
More example studies
- Analyzing an existing datasetA bring-your-own-data study: upload existing responses, code them with a judge, and analyze without re-running trials.
- Framing effects across modelsA matched-pair framing study with decoupling-gap analysis, comparing gain vs. loss framing across three subject models.
- Prompt sensitivity & robustnessSystematic prompt perturbation across six dimensions, with variance decomposition and ranking-stability analysis.
Run this study yourself.
Every walkthrough on this page maps to real platform features. Start a project and follow along.