Factorial & fractional factorial design

Factors are the variables you manipulate (framing, persona, task type) and levels are the discrete values each factor takes (gain vs. loss; expert vs. novice). A full factorial with three two-level factors yields 2×2×2 = 8 conditions; each condition is then replicated many times per model. Full crossing is the default because it estimates every main effect and every interaction without confounding.

The grid grows multiplicatively, and in LLM research every cell costs real API spend. Fractional factorial designs answer this by running a mathematically chosen fraction — one-half or one-quarter for 2-level factors, one-third or one-ninth for 3-level factors — while documenting the aliasing structure: which higher-order interactions become indistinguishable from which main effects. You accept known confounds in exchange for a budget that fits.

Two further strategies complete the toolkit: matched-pair designs, which pair conditions for paired-difference tests, and custom cell selection, which hand-picks only the theoretically interesting combinations from the grid. The right choice depends on whether your question needs the full interaction structure or only targeted contrasts.

AIScholar supports full factorial, fractional factorial (2- and 3-level factors, 3–7 fractionated factors, with aliasing reports), matched-pair, and custom crossing strategies, with replication counts and per-cell cost estimates before any trial runs.

• Experimental design features
• Walkthrough: a 3×3 persona × task factorial

• Matched-pair designPairing conditions on constant content for paired tests and decoupling-gap analysis.
• Variance decompositionPartitioning outcome variance into condition, model, prompt-variant, and residual components.
• Machine psychologyStudying LLMs as behavioral subjects with experimental methods from psychology.

Browse all 11 glossary terms