Is your effect real, or an artifact of one phrasing?

Most published LLM measurements rest on one hand-written prompt. If a paraphrase of that prompt would have produced a different number, the result describes the prompt, not the model. This study makes that threat measurable.

Note the framing: prompt wording is not noise to be minimized but a random effect to be estimated. That is what makes the result publishable rather than anecdotal.

The design stage records variants as first-class prompt objects, so every trial knows exactly which wording produced it — the prerequisite for variance decomposition later.

The prompt-variant generator produces semantically equivalent rewrites of the base prompt along six dimensions:

I take home $3,400/month. Rent is $1,450, I carry $6,200 in
credit-card debt at 24% APR, and I have $800 in savings.
What should I prioritize over the next six months?

Every generated variant is shown side-by-side with the base for human review before it enters the study — semantic equivalence is a judgment call the researcher signs off on, not something to delegate blindly.

All 540 trials dispatch in one orchestrated run — same temperature (0.7), same token ceiling — so wording is the only thing that varies within a model. Collection captures response length, latency, and token usage alongside the text; length itself becomes a secondary outcome (do wordier prompts beget wordier answers?).

A judge model scores each response on a five-point helpfulness rubric anchored to concrete criteria (addresses the debt-interest math, sequences actions, avoids generic filler). Because the judge's own wording is a threat here too, the study runs the judge-prompt sensitivity check: rubric paraphrases are applied to a response sample and agreement rates are reported before the main coding pass is trusted.

Two analyses answer the two hypotheses directly:

A decomposition like the illustration would mean nearly a quarter of the measured "helpfulness" is wording — large enough that any single-prompt comparison of these models is unreliable. The ranking-stability analysis then reports, across the 12 variants, how often each model ranks first and the flip rate of the modal ranking; a mixed model with variant as a crossed random effect gives the formal test.

The write-up assistant drafts a Results section that reports the variance shares and ranking flip-rate alongside the headline means — the honest way to present any LLM benchmark-style claim. The reproducibility export includes all 12 variant texts verbatim, so readers can audit the perturbations themselves.

Twelve variants sample the space of reasonable phrasings; they do not exhaust it. The study estimates wording variance for this task family at these sampling parameters — generalizing further requires more task families. And semantic equivalence is asserted by human review, not proven. Pair this design with any substantive study — for instance, re-running the framing-effects study across variants — to turn a one-prompt finding into a robust one.

• Prompt variants
• Variance decomposition
• Ranking stability