Six studies you could run this week.
Each walkthrough follows one realistic research question through the full eight-stage pipeline — hypothesis, design factors, prompt snippets, execution settings, coding schemes, analysis choices, and write-up. The designs are real and runnable; the sample results are clearly labeled as illustrative.
- Framing effects across models“Do gain- vs. loss-framed dilemmas shift models' stated choices — and do stance and justification move together?”A matched-pair framing study with decoupling-gap analysis, comparing gain vs. loss framing across three subject models.Matched-pair designDecoupling gapGLMM
- Prompt sensitivity & robustness“Is a measured effect robust to how the prompt is worded, or an artifact of one specific phrasing?”Systematic prompt perturbation across six dimensions, with variance decomposition and ranking-stability analysis.Prompt variantsVariance decompositionRanking stability
- Psychometric assessment of an LLM“Can a validated human self-report instrument be administered to a model — and do the scores mean anything?”Administering a risk-attitude scale to models with reliability checks, contamination probes, and a validity checklist.Cronbach's αAlternate formsContamination checks
- Persona × task factorial study“Does an assigned persona change a model's advice — and does the effect depend on the kind of task?”A 3×3 full factorial built with the snippet library and variation grid, analyzed with factorial ANOVA.Full factorialSnippet libraryFactorial ANOVA
- Coding open-ended responses“How do you turn free-text model responses into analyzable data you can actually defend?”LLM-as-a-judge coding with a versioned scheme, multi-judge agreement statistics, and human gold-standard review.LLM-as-a-judgeKrippendorff's αHuman review
- Analyzing an existing dataset“You already collected model responses elsewhere — can you still use the coding and analysis pipeline?”A bring-your-own-data study: upload existing responses, code them with a judge, and analyze without re-running trials.BYOD uploadJudge codingChi-square
How to read these walkthroughs
Every walkthrough is written the way you would actually build the study in AIScholar: the same stages, the same design vocabulary (factors, levels, crossing strategies, snippets), and the same analysis names you will find in the platform. Where a page shows results tables or test statistics, they are illustrative — they show the shape of the output, not findings from a live run. Model behavior shifts across versions, so any real study should collect its own data.
All six studies respect the platform's real constraints: subjects are language models reached through OpenRouter, each trial is a single text prompt and a single text response, and every outcome is something a judge or parser can read off that response. If you want the deeper capability tour first, start with the features overview.
Guides for your field
- Social & management scholarsBring experimental rigor to questions about framing, decision-making, and judgment in language models.
- Psychology researchersAdapt validated instruments, run psychometric checks, and study machine behavior with familiar methods.
- AI-safety & behavioral researchersMeasure refusals, sycophancy, and robustness with reproducible, statistically defensible methods.
- Educators & methods instructorsTeach the full arc of experimental research — design, execution, analysis, write-up — without an IRB or participant pool.
Pick a study and make it yours.
Clone the design vocabulary from any walkthrough into a new project and adapt it to your question.