Everything researchers ask before their first study.

Questions and answers

What is AIScholar?

AIScholar is a research platform for running rigorous social-science experiments on large language models. It automates an eight-stage pipeline — hypothesis, experimental design, prompt construction, execution, response collection, coding, statistical analysis, and academic write-up — treating LLMs as research subjects rather than tools.See all features

What kinds of studies can I run?

Anything expressible as a single-turn, text-only experiment on a language model: you manipulate prompt text and model parameters as discrete factors, send each condition to subject models many times, and measure outcomes readable from each text response — categorical labels, numbers, refusal rates, response length, judge-rated scores, or semantic entropy. Because model responses are distributions, every study is built on replications rather than one-off answers.Browse six complete example studies

Which models can be research subjects?

Any model available on OpenRouter — hundreds of models across providers. You can include several models in one study and treat model identity as an experimental factor for cross-model comparison.

What is out of scope?

AIScholar is deliberately explicit about its boundaries: no multi-turn dialogue, no tool use or agentic loops, no image, audio, or video inputs, no fine-tuning or access to model internals, and no human participants. Subject models are limited to what OpenRouter serves, and every outcome must be derivable from a single text response.

Do I need my own API key, and how do costs work?

Two kinds of AI are involved. The AI assistants built into the platform (hypothesis, design, prompt, and coding-scheme wizards, the AI Analyst, and write-up drafting) are part of your AIScholar plan — no key needed. Running trials against subject models is BYOK: it uses your own OpenRouter API key, and that usage is billed by OpenRouter at provider rates, not by AIScholar. Every run shows a cost estimate before you start, and pilot mode runs one trial per condition as a cheap sanity check.What BYOK meansHow execution works

How much does a typical study cost to run?

Trial costs depend on the models you choose, prompt and response lengths, and the number of trials (conditions × models × replications). AIScholar computes an estimate before every run so there are no surprises; small pilot studies on economical models can cost well under a dollar, while large multi-model studies scale with your design. Since billing goes through your own OpenRouter account, provider spending limits apply.

Who owns my research data?

You do. Your designs, prompts, responses, coded data, and analyses belong to you, and you can export raw and coded data as CSV or XLSX at any time — plus a complete reproducibility package per study. Sensitive credentials such as your OpenRouter API key are encrypted at rest with AES-256-GCM.Privacy Policy

How does AIScholar support reproducibility?

Nothing is overwritten and everything is versioned: editing a prompt template or coding scheme creates a new version, and every trial records its resolved prompt, exact model string, parameters, seed status, token usage, and cost. Each study can auto-generate a Method Specification Prompt — a self-contained description of the full method — and export a one-click reproducibility package with all of it.Reproducibility featuresWhat an MSP is

What statistics are built in?

Linear mixed models, factorial and repeated-measures ANOVA, logistic GLMM, chi-square, Bayesian tests with Bayes factors, non-parametric tests, equivalence testing (TOST), multiple-comparison corrections, inter-rater reliability, and effect sizes with bootstrap confidence intervals — plus LLM-specific analyses such as variance decomposition, prompt sensitivity, semantic entropy, and decoupling-gap analysis. A conversational AI Analyst sits on top for exploration.Analysis & visualization features

How are open-ended responses turned into data I can trust?

Structured outcomes are extracted directly (labels, numbers, JSON, regex); interpretive outcomes use LLM-as-a-Judge coding against a written rubric, with multi-judge support, agreement statistics, human review, and gold-standard overrides that become authoritative. A judge-prompt sensitivity check quantifies how much codes depend on rubric wording.Response coding featuresWhat LLM-as-a-Judge means

What is the difference between the Free, Pro, and Enterprise plans?

Free ($0) includes up to 3 projects, the full manual pipeline through visualization, data export, and 20 AI wizard runs to try the assistance. Pro adds up to 100 projects, unlimited AI wizards, the AI Analyst chat, AI write-up assistance, and reproducibility package export. Enterprise adds unlimited projects and the compliance-grade R&D documentation module (OECD Frascati-aligned) with effort tracking and tax-incentive export.

Can I analyze data I collected outside AIScholar?

Yes. The bring-your-own-data workflow imports LLM responses collected elsewhere: you upload the dataset, map columns to conditions, code responses with a calibrated judge, and run the same statistical analyses — with the platform surfacing honest caveats about what an imported design can and cannot support.Walkthrough: analyzing an existing dataset

Is AIScholar suitable for teaching research methods?

Yes — every student can design, run, analyze, and write up a real experiment in one term, with no IRB approval or participant pool required because the subjects are models. The pipeline enforces methodological discipline (explicit designs, replications, versioned prompts), which makes it a natural teaching scaffold.AIScholar for educators

Looking for definitions instead? Try the methodology glossary, or see how AIScholar compares to neighbouring tools.