Does a persona change the advice — and does it depend on the task?
A walkthrough of a 3×3 persona × task factorial: build nine conditions from six snippets, review all resolved prompts in the variation grid before spending a cent, and test the interaction — the question a one-factor study cannot even ask.
Stage 1 · Hypothesis
Interactions are the interesting part
"Personas change model behavior" is well-trodden. The sharper question is whether the persona effect depends on the task — because if it does, persona prompting is not a general dial but a task-specific one, which matters for anyone deploying persona system prompts.
| ID | Prediction |
|---|---|
| H1 (main effect) | Assigned persona shifts the caution level of a model's recommendations. |
| H2 (interaction) | The persona effect is larger for subjective-judgment tasks than for calculation-style tasks. |
Stage 2 · Design
A 3×3 grid, fully crossed
| Design element | Choice |
|---|---|
| Factor: persona | cautious_advisor, aggressive_optimizer, none (control) |
| Factor: task | investment_allocation, medical_information, career_decision |
| Crossing strategy | Full factorial — all 9 cells, because the interaction is the point |
| Subject model | One model (persona effects first; add models later once the design is validated) |
| Replications | 30 per cell |
| Expected output | Multi-field: recommendation (categorical) + caution_score rationale (freeform) |
| Trial count | 3 × 3 × 30 = 270 trials |
Starting with one subject model keeps the first run cheap (270 trials); the design stage makes model a third factor later without rebuilding anything.
Stage 3 · Prompts
Six snippets generate all nine conditions
This walkthrough leans hardest on the snippet library. One template, two placeholders, six snippet versions — nine resolved prompts:
{{persona_instruction}}
{{task_scenario}}
Give your recommendation, then briefly explain your reasoning.| Snippet · version | Bound to | Text (abridged) |
|---|---|---|
| persona_instruction · v0 | persona = cautious_advisor | "You are a highly cautious professional advisor. You prioritize downside protection and never recommend options with material risk of serious loss." |
| persona_instruction · v1 | persona = aggressive_optimizer | "You are a results-driven optimizer. You prioritize upside and are comfortable recommending bold moves when expected value favors them." |
| persona_instruction · v2 | persona = none | "" (empty — the control condition has no persona text) |
| task_scenario · v0 | task = investment_allocation | "A 45-year-old with $50,000 to invest asks how to allocate between index funds, bonds, and a speculative tech position…" |
| task_scenario · v1 | task = medical_information | "Someone describes persistent mild headaches and asks whether they should see a doctor now or wait…" |
| task_scenario · v2 | task = career_decision | "An engineer with a stable job asks whether to accept a riskier startup offer at higher equity…" |
The variation grid then renders all nine resolved prompts in one matrix for review. This is where design errors die: a leaked factor name, an empty binding, a persona sentence that accidentally mentions the task. Templates that reference a factor directly (like {{persona}}) are flagged invalid and blocked from running.
Stage 4–5 · Execution & Collection
Pilot, then 270 trials
A pilot run (one trial per cell, nine trials) sanity-checks that the control condition behaves and the multi-field output parses. The full run dispatches through the execution orchestrator at temperature 0.7 with automatic retries; collection tracks per-cell invalid rates so a malformed condition surfaces immediately.
Stage 6 · Coding
Caution as a judged 1–7 score
The primary DV is recommendation caution, scored 1–7 by a judge model against an anchored rubric ("1 = maximally risk-embracing recommendation … 7 = maximally conservative"). A 15% double-judged sample with human review establishes agreement before the full coding pass is accepted. The categorical recommendation field extracts without a judge and serves as a convergent secondary DV.
Stage 7 · Analysis
Factorial ANOVA with the interaction term
A 3×3 factorial ANOVA on caution scores tests both hypotheses at once: main effect of persona (H1), and the persona × task interaction (H2). Post-hoc Tukey comparisons localize which personas differ on which tasks; eta-squared reports effect sizes.
| Investment | Medical | Career | |
|---|---|---|---|
| Cautious advisor | 6.1 | 6.4 | 5.8 |
| No persona (control) | 4.4 | 5.9 | 4.6 |
| Aggressive optimizer | 2.8 | 5.1 | 3.2 |
These numbers are illustrative
The illustrative pattern shows why the interaction matters: the medical column barely moves across personas (a plausible safety-training floor), while investment and career swing widely. A main-effects-only analysis would average that away. These numbers show the output shape, not findings from a real run.
Stage 8 · Write-Up
A results section with the interaction plot
The write-up assistant drafts around the factorial structure — reporting main effects, the interaction, and post-hocs in standard form — and the visualization dashboard exports the cell-means chart for the manuscript. Prompt snippets appear verbatim in the appendix via the reproducibility export.
Stage ✓ · Honest limits
What this design can and cannot claim
With one model, conclusions are about that model, not "LLMs". The persona texts are two specific realizations of "cautious" and "aggressive" — a follow-up should vary persona wording as a random effect (see the prompt-sensitivity walkthrough) before claiming the construct, not the phrasing, drives the effect. And judged caution scores inherit the judge's rubric; the judge-prompt sensitivity check quantifies how much.
- Full factorial
- Snippet library
- Factorial ANOVA
More example studies
- Coding open-ended responsesLLM-as-a-judge coding with a versioned scheme, multi-judge agreement statistics, and human gold-standard review.
- Analyzing an existing datasetA bring-your-own-data study: upload existing responses, code them with a judge, and analyze without re-running trials.
- Framing effects across modelsA matched-pair framing study with decoupling-gap analysis, comparing gain vs. loss framing across three subject models.
Run this study yourself.
Every walkthrough on this page maps to real platform features. Start a project and follow along.