You already have the responses. Now what?
A walkthrough of a bring-your-own-data project: upload a CSV of model responses collected elsewhere, map its columns to conditions, run the same judge-coding and statistical machinery as a native experiment, and know exactly which claims the imported data can and cannot support.
Stage 0 · Starting point
A dataset from outside the platform
Not every study starts at Stage 1. Suppose a colleague hands you 1,800 model responses from a customer-support benchmark: two system-prompt conditions ("empathetic" vs. "efficient"), three models, 300 responses each, collected with their own scripts months ago. The collection is done; the measurement and analysis are not. AIScholar's BYOD mode exists for exactly this case.
| Column | Contents | Mapped to |
|---|---|---|
| response_text | The model's full reply | Trial response body |
| system_condition | "empathetic" | "efficient" | Condition factor 1 |
| model_name | Three model identifiers | Condition factor 2 |
| scenario_id | 12 support scenarios | Condition factor 3 (grouping variable) |
| collected_at, temperature | Collection metadata | Trial metadata (preserved verbatim) |
Stage — · Import
Upload, map columns, verify structure
A BYOD project skips Stages 1–4 entirely — the platform blocks them, because pretending imported data came from a designed experiment would be dishonest bookkeeping. The upload flow ingests the CSV, you map columns to response text and condition labels, and the platform builds a structure summary: cells, counts per cell, and any imbalance.
You also attach a free-text dataset context note — how the data was collected, by whom, with what sampling parameters. Every downstream AI assist (coding suggestions, analysis interpretation, write-up drafting) reads that context, so the assistance stays observational: this is found data, and the platform's prose treats it that way.
Stage 6 · Coding
The same judge pipeline as native experiments
From here the project behaves like any other. A versioned coding scheme defines two variables — empathy_display (1–5, anchored rubric) and resolution_offered (boolean) — and an LLM judge codes all 1,800 responses on your OpenRouter key, with a human-reviewed calibration sample and a second judge on a 15% overlap. (For the full calibration discipline, see the judge-coding walkthrough.)
Stage 7 · Analysis
Chi-square, mixed models, and honest caveats
The analysis stage treats the imported cells as conditions: a chi-square test on resolution rates by system condition, and a mixed model of empathy scores with scenario as a random effect.
| Comparison | Statistic | Value |
|---|---|---|
| Resolution rate: empathetic vs. efficient | χ²(1) | 4.7, p = .030 |
| Empathy score by condition | Mixed-model b | 1.31 (SE 0.12) |
| Condition × model interaction | LRT | χ²(2) = 9.8, p = .007 |
These numbers are illustrative
With found data, the interaction row is the one to treat most carefully: if the original collection did not randomize scenario order or hold sampling parameters constant across models, condition differences may be confounded with collection differences. These values show the output shape, not findings from a real run.
Stage 8 · Write-Up
Drafts that say 'observational' out loud
The write-up assistant drafts around the dataset context: the Method section describes the external collection procedure as reported, flags what is unknown (exact sampling parameters, retry behavior), and keeps causal language out of the Results. The reproducibility export bundles the original CSV, the column mapping, coding scheme versions, and every judge call — so the analysis, at least, is fully re-runnable even though the collection is not.
Stage ✓ · Honest limits
What imported data can and cannot support
BYOD projects inherit every weakness of the original collection: unknown prompt variation, unlogged retries, model versions that may have changed since. The platform's job is to make the measurement layer (coding, agreement, statistics) as rigorous as a native study while keeping the provenance boundary visible. When the found-data analysis turns up something interesting, the natural next step is a designed replication — start with the factorial walkthrough and collect fresh data under controlled conditions.
- BYOD upload
- Judge coding
- Chi-square
More example studies
- Framing effects across modelsA matched-pair framing study with decoupling-gap analysis, comparing gain vs. loss framing across three subject models.
- Prompt sensitivity & robustnessSystematic prompt perturbation across six dimensions, with variance decomposition and ranking-stability analysis.
- Psychometric assessment of an LLMAdministering a risk-attitude scale to models with reliability checks, contamination probes, and a validity checklist.
Run this study yourself.
Every walkthrough on this page maps to real platform features. Start a project and follow along.