Prompt sensitivity

Any single prompt is one arbitrary point in a large space of equivalent phrasings. A finding measured with one phrasing might be a property of the model — or an artifact of that particular wording. Prompt-sensitivity analysis makes the distinction empirical: generate semantically-equivalent variants of the prompt, run the experiment across all of them, and measure how much results move.

Perturbations fall into recognizable dimensions: lexical paraphrasing (synonym swaps, restructured sentences), structural reformatting (list vs. prose, section order), verbosity (concise vs. expansive), output-format requests (JSON vs. Markdown), instruction specificity (adding or removing constraints), and persona framing (assigned role or style). Systematic variation along these dimensions turns "wording" from a lurking confound into a measured random effect.

Useful summary metrics include the performance spread across variants, the share of variance attributable to wording (via variance decomposition), the stability of model rankings across variants, and the flip-rate of the modal response. A robust effect survives paraphrase; a fragile one is a caveat that belongs in the write-up.

AIScholar auto-generates prompt variants along six perturbation dimensions, runs them as a factor or random effect, and reports performance spread, variance attribution, and ranking-stability analyses out of the box.

• Prompt construction features
• Walkthrough: prompt-sensitivity study

• Variance decompositionPartitioning outcome variance into condition, model, prompt-variant, and residual components.
• Semantic uncertaintyShannon entropy over meaning-clustered responses — response consistency as a measurable outcome.
• Method Specification Prompt (MSP)A self-contained method specification precise enough for independent re-implementation.

Browse all 11 glossary terms