From research scores to welfare estimates

1. The key finding

No numerical welfare calculation produced the current 7.4 or 6.8 scores. A model read paper material and assigned each number against a verbal rubric. The site can reproduce the ordering and the separate weighted arithmetic used on the main evaluation dashboard, but it cannot reconstruct 7.4 from a chain of policy influence, outcome changes, and welfare weights because that calculation was never performed or recorded.

This explains how a paper described as having a “strong but not exceptional” welfare case could rank second: it was second within an incomplete 41-paper subset, and only one paper in that subset scored above 7.4. That rank does not establish global superiority over the other 1,178 records or over research outside the collection.

2. How the displayed numbers were obtained

The columns are different quantities. Rank is an ordering within the assessed subset. The weighted evaluation score helps choose which papers might benefit from evaluation. The earlier holistic score is another model judgment. None of the four papers had a published human impact-category rating contributing to its impact rank.

3. What the examples show

The stored explanations identify plausible pathways, but key causal and distributional quantities remain unknown. Open each example for the evidence credited and the reason its numerical placement remains uncertain.

4. A proposed welfare calculation

The calculation should follow the causal chain from research to changed choices, then to outcomes, then to welfare. For research that changes a funding allocation:

For a rule or technology without a useful budget measure:

The counterfactual matters at every step. The research-caused change is relative to what would happen without the paper. For reallocations, the relevant return is the difference between the funded option and the displaced alternative. For large reallocations, integrate over a funding-response curve rather than assuming a constant return. Under uncertainty, estimate the expectation of the whole product because uptake, reach, effectiveness, and cost can be correlated.

Research impact and evaluation value need separate counterfactuals

A paper’s research contribution compares decisions with its findings or tools against decisions without them. The value of commissioning an evaluation compares publication and use of that paper with an evaluation against publication and use without one. An evaluation may correct errors, reduce misplaced confidence, improve methods, or accelerate warranted uptake. Neither the current impact score nor the dashboard’s weighted evaluation formula computes that value.

5. Distributional and moral assumptions

Income and consumption

Use real consumption per person on a common purchasing-power and price-year basis when possible, and apply the calculation over the beneficiary distribution rather than a country average. For curvature η, compare log utility (η = 1) with more curved alternatives. The table uses hypothetical annual consumption of $1,000 and $50,000.

Under log utility, the same proportional gain has equal value at either consumption level; an equal dollar gain is already valued more for the poorer person. Greater curvature favors the poorer recipient in both comparisons. A sensible pilot would show η = 1, 1.5, and 2 side by side. Using 1.5 as a central scenario would be an exploratory assumption, not an established empirical coefficient or a settled moral preference.

6. What the pilot found

We have now run the proposed six-paper pilot and applied the same protocol at screening depth to 18 more papers. None supports a defensible total expected-welfare estimate with the evidence currently assembled. Four pilot cases do support useful conditional calculations, and the wider screen makes the missing research-to-decision links explicit.

1. Identify the actual decision and the without-paper alternative.
2. Record reachable scale, the change attributable to the research, outcome effects, beneficiary circumstances, duration, costs, displacement, and plausible harms.
3. Tag each input as source estimate, analyst assumption, or unknown, with units and provenance.
4. Report natural outcomes first, then welfare results under several named scenarios.
5. Show which uncertain parameters reverse the ranking. Preserve “needs evidence” where a numerical total is not defensible.
6. Compare those outputs with historical human priorities and commissioned evaluations, looking for interpretable patterns by beneficiaries, field, method, outcomes, observed versus hypothetical behavior, neglectedness, and decision pathway.

Explore the six detailed cases and 18-paper expansion, including the input provenance, low/central/high scenarios, natural outcomes, separate evaluation-value judgments, and downloadable JSON.

Historical choices can help reveal useful heuristics, but they should not be treated as ground-truth welfare labels. Selection reflects changing strategy, availability, evaluator capacity, and the value of evaluation as well as expected research impact. Comments may clarify those mechanisms better than a score alone.

The public 0–10 numbers should remain labeled as ordinal model assessments until a transparent mapping to quantified outcomes is specified and tested. Bands or whole numbers would better match the present precision.

Deferred viewer controls

A later version could let viewers adjust consumption curvature, consumption-to-health conversion, species welfare ranges and sentience beliefs, future-welfare assumptions, and risk attitudes. Personal settings should change that viewer’s comparison without overwriting shared evidence or ratings. Empirical disagreement, welfare-capacity uncertainty, and moral preference should remain visibly distinct.

7. Learning from past human prioritization

A separate, privacy-safe analysis asks what The Unjournal’s recorded priorities and evaluation choices reveal about its past practice. The source snapshot contains 325 Coda prioritization records, including 238 with at least one human rating and 57 that became published evaluations. A terminal-outcome model uses 194 rows whose current status can conservatively be treated as either selected or deprioritized/blocked; pending cases are excluded.

The initial descriptive patterns are suggestive rather than rules. For example, 4 of 17 terminal records coded as wealthy-country work were selected, compared with 32 of 73 coded as lower- or middle-income-country work. Papers coded as using observed choices were selected in 9 of 19 cases, compared with 2 of 7 for hypothetical preferences. The cells are small, the coding is preliminary, and topic, methods, publication status, timing, evaluator availability, and other constraints are confounded.

A five-fold model using the prior human rating alone distinguished the terminal outcomes reasonably well in this sample (AUC 0.80; Brier score 0.174). Adding the currently available metadata did not improve out-of-sample performance (AUC 0.769; Brier score 0.190). This is a useful warning against turning a few descriptive correlations into scoring rules.

This exercise does not recover moral weights or a welfare function. Historical ratings bundle research impact, value of an additional evaluation, credibility, timing, open-access and author constraints, evaluator supply, and changing organizational priorities. The model can describe choices and generate hypotheses for review; income curvature, cross-species weights, future-life assumptions, and other moral choices must remain explicit and separate.

Read the modeling and validation plan · Download the privacy-safe aggregate results