Matched-pair design

Scenarios differ from one another in ways that add noise: some dilemmas are harder, some vignettes more emotionally loaded. When each scenario serves as its own control — appearing once in each framing — that between-scenario variation cancels in the paired difference, giving the test substantially more power than an unpaired comparison of the same size.

Matched pairs also unlock a distinctive LLM analysis: the decoupling gap. When a manipulation flips a model's stated stance, does its justification change too, or does the model produce the same reasoning while reversing its conclusion? Measuring stance and justification separately across the members of each pair reveals whether the two move together or come apart — a question that only makes sense with paired data.

The design suits any manipulation naturally expressed as a minimal contrast on constant content: framing effects, politeness manipulations, persona swaps, or equivalence tests between two phrasings of the same instruction.

AIScholar supports matched-pair as a first-class crossing strategy and ships a decoupling-gap analysis that tests whether stance and justification shift together or independently across pair members.

• Experimental design features
• Walkthrough: framing effects across models

• Factorial & fractional factorial designFull vs. fractional crossing of factor levels, aliasing trade-offs, and when to use each.
• Prompt sensitivityMeasuring how results change under semantically-equivalent rewordings of a prompt.
• Variance decompositionPartitioning outcome variance into condition, model, prompt-variant, and residual components.

Browse all 11 glossary terms