Variance decomposition

An observed effect is only as interesting as the share of variation it explains. If conditions account for 40% of outcome variance, the manipulation is doing real work; if prompt wording accounts for more variance than the conditions do, the study is chiefly measuring phrasing artifacts. Decomposing variance makes these proportions explicit instead of leaving them buried in a significance test.

In LLM research the decomposition has a characteristic structure: condition (the factors you designed), model (when several subject models are included), prompt variant (when semantically-equivalent paraphrases are run as a random effect), and residual (the stochastic draw-to-draw variation of sampling). Each component answers a different question — is the effect real, is it model-specific, is it wording-fragile, and how noisy is the measurement.

The technique pairs naturally with mixed-effects models, which estimate the same structure inferentially: fixed effects for designed factors, random effects for items and prompt variants. Reporting variance components alongside p-values gives readers the magnitude context that null-hypothesis tests alone omit.

AIScholar runs variance decomposition as a built-in analysis, partitioning outcome variance into condition, model, prompt-variant, and residual components, and supports linear mixed models with crossed random effects for the inferential counterpart.

• Built-in statistical analyses
• Walkthrough: prompt-sensitivity study

• Prompt sensitivityMeasuring how results change under semantically-equivalent rewordings of a prompt.
• Semantic uncertaintyShannon entropy over meaning-clustered responses — response consistency as a measurable outcome.
• Factorial & fractional factorial designFull vs. fractional crossing of factor levels, aliasing trade-offs, and when to use each.

Browse all 11 glossary terms