Machine psychology

The core move of machine psychology is to treat the model as a participant rather than a tool. Instead of asking whether a model can complete a task, the researcher manipulates the input systematically — framing, persona, wording, parameters — and measures how the distribution of responses shifts. A single answer from a model is one draw from a probability distribution, so machine-psychology studies rely on many replications per condition rather than one-off outputs.

This framing imports the discipline of experimental psychology: hypotheses stated before data collection, factorial manipulation of independent variables, operationalized outcome measures, inferential statistics, and effect sizes. It also imports psychology's hard lessons — instruments must be validated for the new population, measurements must be reliable, and results must survive robustness checks such as paraphrasing the prompt.

Machine psychology differs from benchmark evaluation. A benchmark asks "how well does the model score on task X?"; a machine-psychology experiment asks "what changes the model's behavior, by how much, and does the effect generalize across models and phrasings?" The unit of analysis is the trial — one model, one fully-resolved prompt, one parameter set, run once — and the object of interest is the pattern across thousands of trials.

AIScholar is built for machine psychology end to end: an eight-stage pipeline takes a study from hypothesis and factorial design through prompt construction, replicated execution against any OpenRouter model, response coding, statistical analysis, and academic write-up.

• The full research pipeline
• How psychologists use AIScholar

• Factorial & fractional factorial designFull vs. fractional crossing of factor levels, aliasing trade-offs, and when to use each.
• Prompt sensitivityMeasuring how results change under semantically-equivalent rewordings of a prompt.
• Psychometric validity for LLMsRe-establishing reliability and validity when human instruments are administered to models.

Browse all 11 glossary terms