Factorial experiment design, built for language models.

Every AIScholar study begins with a hypothesis: a falsifiable statement about how a language model will behave under specific conditions. The platform treats LLMs as research subjects — you send a model a prompt, record its response, and analyze the distribution of responses across many replications, exactly as an experimental psychologist studies human participants.

The hypothesis stage gives you a structured place to state your prediction, its rationale, and the expected direction of the effect. An AI wizard acts as a sounding board: it can brainstorm candidate hypotheses from a research area, sharpen a vague question into a testable one, and flag predictions that the platform cannot actually measure — for example, anything that would require multi-turn dialogue, tool use, or access to model internals. Feasibility is checked up front, before you invest in a design.

A design is built from factors (the variables you manipulate, like "framing") and levels (the discrete values each factor takes, like gain-framed vs. loss-framed). How those levels combine into experimental conditions is your crossing strategy — and AIScholar supports four:

• Full factorial

  Every combination of every factor's levels. The default when you want all main effects and interactions and the cell count is manageable.

• Fractional factorial

  A principled subset of the grid for 3–7 factors at 2 or 3 levels, with 1/2, 1/4, 1/3, or 1/9 fractions. AIScholar reports the aliasing structure so you know exactly which effects are confounded.

• Matched-pair

  Pairs conditions for paired-difference tests — the same scenario under two framings — enabling decoupling-gap analysis of whether a model's stance and justification move together or apart.

• Custom cells

  Hand-pick specific conditions from the grid instead of running the whole product, when only certain combinations are theoretically interesting.

1. 1

   Define factors and levels

   Name each variable you plan to manipulate and its discrete levels. Levels are analytical labels — the actual prompt wording lives in the snippet library, so the design stays clean and the text stays editable.

2. 2

   Choose a crossing strategy

   Full factorial, fractional, matched-pair, or custom. The design view shows the resulting condition grid and cell count so you can see the cost implications immediately.

3. 3

   Declare the expected output

   Specify what shape each response should take — freeform text, a categorical label, a number with a valid range, a JSON schema, a regex-extracted value, or multiple fields at once. This drives automatic extraction and validation later.

4. 4

   Set models and replications

   Pick subject models (model identity can itself be a factor) and how many replications each condition gets. Power-aware defaults nudge replication counts high enough that your analysis can actually detect the effect you predicted.

5. 5

   Let the AI wizard propose, then review

   The design wizard can draft a complete design from your hypothesis — factors, levels, crossing, outputs. You always review and edit before anything runs; the wizard proposes, you decide.

• Methodological warnings

  Rule-based, contextual warnings flag issues like underpowered cells, unbalanced designs, or missing controls — citing the methodological literature.

• AI design assistance

  Wizards understand both the platform's constraints and your project context, so proposed designs are runnable, not aspirational.

• Design versioning

  Designs are recorded with the project so every downstream trial, coding pass, and analysis traces back to the exact grid that produced it.

Once your design is set, the next step is realizing each condition as concrete prompt text — that's the job of prompt construction.