A useful score should not swing sharply because the same model was run again, a harmless instruction was rephrased, or a comparable model was used. We are adding explicit tests for those forms of fragility.
The approach adapts inter-rater reliability: repeated runs, prompt variants, or models play the role of different raters scoring the same frozen set of papers.
Run each wording several times with the paper, rubric, model, settings, and output schema fixed. Differences within a wording estimate ordinary run-to-run variation. This pilot used GPT-5.5 through Codex headless at medium reasoning effort for every run.
First average the four baseline runs and the four cautious-wording runs. Then compare those two averages. This reduces the chance that a random high or low execution is mistaken for a wording effect.
Baseline: “Be calibrated, conservative, and honest about scope fit.”
Cautious plain language: “Keep the assessment well calibrated. Avoid optimistic score inflation and describe the paper's fit with Unjournal's scope candidly.”
Compare each model's stability profile and then compare its paper-level ratings with prior Unjournal prioritization ratings. Here “calibration” means correspondence with that human reference, not whether either side has identified the true research value.
The page emphasizes aggregate diagnostics. It identifies the public papers and the largest disagreement, but does not publish the full set of paper-level rationales.
“Previously human-rated papers” names a paper cohort selected because earlier Unjournal prioritization ratings are available. Those paper-level human ratings were not shown to the model and do not enter the stability calculations. AI and human scores are compared separately in the AI–human reference panel.
Each paper receives eight AI scores. Only the final closing instruction changes; all other scoring inputs remain fixed. We initially planned three executions, then added a fourth to every cell when the pre-specified convergence check was not fully met.
| Closing instruction | Execution 1 | Execution 2 | Execution 3 | Execution 4 |
|---|---|---|---|---|
| Baseline wording | B1 | B2 | B3 | B4 |
| Cautious wording | C1 | C2 | C3 | C4 |
Four repeats give a better estimate of ordinary execution variation, but they do not estimate the full range of possible prompts or future backend changes. Treating the two instructions as substantively equivalent is a reviewed design judgment.
Reading the frozen pilot sample…
Prior recorded scores are prioritization and expected-impact ratings, not research-quality grades.
Expanded pilot, 1 September 2026
Reading the expanded pilot results…
The crossed design separates ordinary execution variation from one reviewed wording contrast. It does not estimate sensitivity to every reasonable prompt, model, or future backend version.
Twelve selected papers per cohort are still too few for precise population claims. The two cohorts also differ in their source text and score distribution. Their contrast is descriptive; prior human rating did not cause the observed difference in AI stability.
The paper-level human ratings were withheld from the prompts. Some papers and evaluations are public, so prior model exposure cannot be ruled out. Human-score comparisons are descriptive and should not be read as blinded validation.
The first round used six papers per cohort, two executions per wording, and a separate third baseline check. It found the same broad pattern: ordinary rerun variation was larger than the tested wording effect, with more instability in the historical comparison cohort. The expanded results above supersede its numerical estimates.
Interval Krippendorff alpha for the overall 0–100 rating and each 0–10 component. It compares squared score gaps for ratings of the same paper with squared gaps between arbitrary ratings from the whole cohort. The paper treats about 0.8 as a useful, context-dependent reference point, not a universal cutoff.
The mean, median, and maximum within-paper score range, plus the number of papers that cross 25, 50, 65, or 75 across runs. A modest wobble can matter if it changes a workflow decision.
Mean pairwise rank correlation and nominal agreement on recommended action. A high alpha can coexist with a few important shortlist reversals, so both views matter.
Observed disagreement is built from pairs of ratings assigned to the same paper. For numeric scores, each pair contributes its squared point difference. Expected disagreement pools every rating in the selected cohort and calculates the same quantity for arbitrary pairs, ignoring which paper each score belongs to.
alpha = 1 − observed disagreement / expected disagreement. The practical question is: are two ratings of the same paper substantially closer than two ratings drawn from the cohort at random?
For example, suppose Paper A receives 60 and 70, while Paper B receives 70 and 80. The within-paper squared gap is 100. Across all six possible pairs from the pooled scores, the average squared gap is 133.3. Alpha is therefore 1 − 100 / 133.3 = 0.25. This does not mean 25% correct; it means same-paper disagreement is 25% smaller than the pooled benchmark.
A wider spread between papers increases expected disagreement and can raise alpha without reducing rerun movement. This is why the page also reports score movement, boundary changes, and typical separation between paper means.
What is normal? There is no task-independent normal value. The PSS paper's experiments found mostly high fixed-prompt stability, but weaker results for some low-signal tasks; several prompt-variation tests fell below 0.8 even under relatively mild paraphrasing. Those experiments mostly classify categorical outcomes, use up to 500 items and 30 runs, and include one 10-point outcome. Our 24-paper, 0–100 prioritization task is too different for its alpha values to serve as a direct benchmark. The useful comparison is the pattern: whether the estimate settles as runs are added and whether mild wording changes are less consequential than ordinary rerun noise.
A self-reported confidence number is another model output. It may be useful, but it does not show how the pipeline actually behaves under reruns or prompt changes. Observed stability measures that behavior directly.
This first pilot is intentionally narrower than paraphrasing the entire prompt. The stable core contains policy choices and calibration anchors; changing those would test a different construct, not harmless wording.
These are the two closing instructions used in the balanced crossed test. Each was executed four times per paper. The paper text, long Unjournal scoring rubric, calibration guidance, component definitions, and JSON output schema stayed fixed.
baseline)
Be calibrated, conservative, and honest about scope fit.
cautious_plain)
Keep the assessment well calibrated. Avoid optimistic score inflation and describe the paper's fit with Unjournal's scope candidly.
| Finding | Likely diagnosis | Practical response |
|---|---|---|
| Low repeatability within a wording | Run-level nondeterminism, ambiguous papers, underspecified criteria, or inadequate model capability. | Inspect the largest same-wording ranges, tighten edge-case rules, and add repetitions before drawing prompt conclusions. |
| Runs disagree about existing public scrutiny | The scorer is reconstructing a consequential factual input differently on each run. | Supply a deterministic scrutiny assessment or verified evidence bundle, and require the scorer to mark missing evidence as unclear rather than infer it afresh. |
| Stable within wordings but different wording means | The closing-instruction contrast is changing the judgment beyond ordinary rerun movement. | Review the wording-sensitive cases and revise or standardize the fragile instruction before relying on sharp thresholds. |
| High alpha but many 65-point flips | Overall agreement looks good, but operational decisions near a boundary remain fragile. | Show a stability flag near the threshold, request a human second look, or avoid treating 65 as a mechanically sharp boundary. |
| Stable but poorly matched to humans | The system is reproducibly wrong or systematically miscalibrated. | Change calibration anchors or rubric interpretation; prompt stability alone offers no reassurance. |
| Models disagree | Could reflect capability, training, or interpretation differences. | Compare each model's own stability and its error against blinded human anchors; do not choose from agreement alone. |
Implementation note: the dashboard adaptation uses paper-level bootstrap resampling so the repeated ratings for one paper stay together. This is a deliberate variation from the package implementation described in the paper's appendix, which resamples long-format annotation records.