Early stageIn use for research discovery and prioritization. Coverage and scoring are still developing. Add evidence, ratings, or corrections to help improve the shortlist; the team makes final evaluation decisions.
Help choose research for independent evaluation by The Unjournal
Independent-rating view. Existing priority scores, recommendations, human feedback, and prioritization analyses are hidden until you rate each paper in this browser, reducing anchoring from other judgments. Abstracts and source metadata remain visible. Papers are ordered newest first.Why rate independently?Return to the full dashboard.
Find research worth evaluating: choose a topic, read the evidence, and add your judgment. Scores estimate evaluation priorityHow strongly we recommend commissioning an independent Unjournal evaluation of this paper. This considers decision relevance, the added value of evaluation, timing, and likely influence. A high score does not mean the research is good or bad. on a 0–100 scale. They are provisional, not ratings of research quality.
How human suggestions reach team review
Public-safe human suggestions appear here with their origin. A qualifying team rating—or sufficiently strong, discussed public feedback—flags the paper for an independent team assessment and adds a Needs further prioritization assessment label. Team members can follow that label to the internal prioritization hub (sign-in required) for an independent follow-up assessment. The label marks a workflow step, not an endorsement.
About this prioritization tool What the scores mean, sources, methodology, and feedback
What is The Unjournal?
We commission and publish independent, public evaluations of research that can inform high-stakes global decisions. We focus on economics, quantitative social science, forecasting, and policy-relevant research—including development economics, global health, animal welfare, AI governance, climate policy, and catastrophic risks.
Learn more →
Early stage, in use. Coverage and scoring depth will improve as we expand sources. Use Rating source
above the list to compare AI-only scores, public human aggregates, or the human-AI synthesis.
Read the preliminary score-stability report (24 selected papers, 192 ratings; an exploratory diagnostic).
Papers are automatically discovered from
multiple academic sourcesCurrently scanning NBER, arXiv, CEPR, EA Forum research links, Semantic Scholar, OpenAlex/SSRN, RePEc, and selected AI-research organizations. New papers are fetched periodically and scored automatically.,
then scored by AI models against Unjournal's prioritization criteria. Paper discovery and scoring are normally refreshed twice weekly; feedback aggregates can refresh up to hourly. See dashboard statistics and the score-stability methods page for more detail.
Prioritization = expected value of commissioning an evaluation, not quality endorsement.
A prominent but flawed paper may score HIGHER than a rigorous but obscure one, because independent evaluation adds more value there.
Two display lenses (toggle above the list)
The list can be scored and sorted through either of two transparent weightings of the six sub-scores — toggle
Evaluation-relevance weighting in the controls, and unfold What is this score? to see the live weights:
Criterion
Research relevance (readers)
Evaluation priority (UJ, default)
Potential decision relevance
40%
30%
Methodological potential
25%
10%
Real-world influence
20%
10%
Prominence / attention
15%
0%
Timing value
0%
20%
Neglectedness / value of evaluation
0%
30%
Human and synthesis ratings: Each rater's latest rating for a paper counts once. Direct 0–100 ratings are used as entered; quick votes map from Strong No through Strong Yes to 10, 30, 50, 70, and 90. Human-only is a weighted mean: a quick rating has weight 1; a full rating without detailed written discussion has weight 2; a public full rating with detailed discussion has weight 3; and a team full rating with detailed discussion has weight 4. “Detailed” currently means at least 120 characters or 20 words. The AI score has weight 3 in the synthesis, measured in the same quick-rating equivalents. These weights are admittedly ad hoc and should be revisited as evidence accumulates. Public submissions are not identity-verified, so treat small human samples as indicative rather than a team decision.
Research relevance ranks work by importance, rigor, and relevance for readers looking for useful research. Evaluation priority (the default) adds timing and value-of-evaluation (neglectedness) — the expected value of commissioning an independent Unjournal evaluation. If a sub-score is missing for a paper, its weight is dropped and the rest renormalized; out-of-scope papers are floored.
Model two-track scoring
Underneath the lenses, the AI model treats prominent and less-prominent work differently. Prominence is not a quality bonus for obscure work; it mainly tells us whether independent evaluation could matter because the work is already influential, or because a neglected but solid piece might otherwise be missed.
Criterion
Prominent work
Less-prominent work
Potential decision relevance
40%
30%
Timing value
25%
15%
Real-world influence
20%
20%
Methodological potential
10%
25%
Prominence / existing attention
5%
Context only
For prominent work (NBER, CEPR, World Bank, top journals), decision-relevance dominates; methodological weakness can make public evaluation more valuable rather than less. For less-prominent work, the data, identification strategy, clarity, and evaluability matter more, because serious flaws make evaluation less likely to change decisions.
Scoring rubrics (0–10 each)
1. Global Decision-Relevance (most important)
9–10: Directly informs active decisions by major funders or policymakers (GiveWell cost-effectiveness, WHO policy, climate treaty design). Specific organizations can be named.
7–8: Addresses a recognized global priority with clear policy implications, but the link to specific decisions is less direct.
5–6: Relevant to global welfare in a general sense. Interesting for the field but specific decision-relevance is moderate.
3–4: Tangentially related to global priorities. Primarily academic interest.
1–2: No clear connection to decisions affecting global welfare.
Field-specific: Development and global-health work should connect to intervention choice, delivery, or scale; AI-governance work should bear on a specific governance or institutional choice rather than merely discuss AI; animal-welfare work should identify the affected population, welfare margin, and plausible decision-maker; climate work should change an emissions, damages, adaptation, or policy-design margin; forecasting and meta-science work should connect accuracy or research practice to a concrete institutional decision.
2. Prominence
9–10: NBER working paper, top-5 journal, Nobel/Clark laureate, >500 citations, major media coverage.
1–2: Unknown author, self-published, no institutional backing.
Note: NBER/CEPR/World Bank/IMF sources are treated as high-attention signals, not proof of quality or publication stage. Some NBER papers are later published in journals, and this should be checked when timing matters. Prominent flawed work can be valuable to evaluate because people may already be using it.
3. Real-World Influence
9–10: Already cited in policy documents, GiveWell/Open Phil analyses, government reports. Named organizations are using this.
7–8: Likely to influence decisions soon. In an active policy debate. Authors have policy connections.
5–6: Could influence decisions if findings hold up. Relevant to active debates but not yet cited.
3–4: Academic contribution with indirect policy relevance.
1–2: Purely academic exercise with no clear path to influence.
4. Timing Value
9–10: Working paper/preprint released in last 6 months. No peer review yet. Authors actively seeking feedback.
7–8: Working paper 6–18 months old. Under review but not yet published.
5–6: Recently published (1–2 years) in a venue where more review would add value. R&R at journal.
3–4: Published 2+ years ago but still influential. Adds transparency but less urgency.
1–2: Old published work with established peer review. Feedback largely moot.
By methodology: RCTs & field experiments benefit most from early feedback (pre-registration, pre-analysis). Policy reports have narrow windows. Theoretical work is less time-sensitive.
5. Methodological Potential
For prominent work: This is a secondary consideration. If it’s prominent and decision-relevant, score 7+ and move on. Quality assessment is for the evaluation stage.
For less-prominent work (the tie-breaker):
9–10: Innovative methodology, strong identification strategy, credible real-world outcome data where possible, reproducible analysis with shared code/data.
7–8: Solid methods appropriate for the research question.
3–4: Methodological concerns that would make evaluation difficult.
1–2: Not really quantitative. Literature review, opinion piece, or purely conceptual.
Field-appropriate standards are illustrative, not mechanical. We should not penalize fields where RCTs are impossible, but observed choices and measured outcomes usually deserve more weight than recall or hypothetical stated-preference evidence when both are available.
Development/health: RCTs, DiD, regression discontinuity, IV
Environmental/climate: Integrated assessment models, panel data, natural experiments
AI governance: Mixed methods, surveys, formal models
Animal welfare: Revealed-preference or behavioral evidence where available; stated preference, DCEs, and welfare calculations where direct evidence is limited
Political science: Quasi-experimental, panel data, surveys
Macro/trade: DSGE, gravity equations, synthetic control
Stored AI screening recommendations
The labels below come from the original AI screening score. The large displayed score uses the selected weighting and rating source, so it can differ from the screening label. Neither is a team decision.
Score range
Recommended action
What it means
75–100
AI shortlist
The model recommends human assessment next. This does not mean the paper has a human rating or that Unjournal has decided to evaluate it.
50–74
Watchlist
Potentially relevant, but not currently a high-priority evaluation candidate.
25–49
Deprioritize
The stored AI screening recommendation is to give other candidates attention first.
<25
Out of scope
Not quantitative social science, or fundamentally outside UJ coverage.
Calibration and stability answer different questions. Calibration asks whether AI ratings line up with human judgments. Stability asks whether the same papers keep similar ratings when the model is run again or a substantively equivalent instruction is reworded. A system can be stable but wrong, so stability is a reproducibility diagnostic rather than an accuracy claim. Our pilot covers 24 selected papers with 192 repeated ratings and remains exploratory; it is not a full-dashboard reliability estimate. Read the report and methods note.
Four-stage pipeline
Suggesting — A paper is suggested (by AI or human) with a 0–100 rating and discussion of relevance
Assessing — A second team member gives an independent rating (without seeing the first)
Voting — If avg rating ≥ 65%, the field group votes (Strong Yes to Strong No)
Evaluation — An evaluation manager commissions 2+ public evaluations via PubPub
Comment directly on this page using the
Hypothes.is sidebar
(look for the < tab on the right edge of the page). Highlight any text and add your annotation —
visible to all Hypothes.is users. You can also use the feedback buttons on each paper card.
Turn on quick rating to add your judgment directly on each card. Preview the abstract, then choose --, -, ~, +, or ++ directly on the card. Written comments and a percentile rating can be added from Details / rate/discuss. Revisiting a paper from the same browser updates that rating rather than adding another.
-
Shown
-
AI shortlist
More filters and source controls
Targeted intake:Requested topic-focused searches deliberately oversample some areas; inclusion is not an endorsement or prevalence estimate.Browse the search catalog →
Score weightsCustom weights
Adjust these sliders to create your own priority score, then sort by Custom weights. This changes only your browser view; it does not change The Unjournal scorer.
Quick-rate mode: for each paper, how strongly should The Unjournal commission an evaluation?
(-- Strong No … ++ Strong Yes)For aggregation and sorting, the five quick ratings translate to percentile-equivalent scores: -- = 10, - = 30, ~ = 50, + = 70, and ++ = 90. These are deliberately spaced away from 0 and 100..
Preview the abstract before rating (hover over its preview line, or open Details / rate / discuss for more context); title-only ratings are not useful.
Read the paper or linked source when practical. Each click is saved to The Unjournal's server, and a later choice from the same browser revises that paper's rating. Add your name or email once below; the detailed form will reuse it. To avoid being anchored by existing scores, use the independent-rating view. Rating and privacy guide.0 rated this session
Scores combine AI estimates with human ratings where available. Small score differences are not decisive. Open a paper for the evidence and rating breakdown.
Suggest research for this list
Know of high-impact research we should consider for evaluation — or other work this tool should be doing?
Leave an email if you're open to follow-up discussion; if we later introduce compensation for useful
contributions, earlier contributors will be grandfathered in.
Is this your own research? This form is for suggesting others’ work.
To submit your own work (or your organisation’s) for evaluation, please
use the author submission form ↓.
On The Unjournal team? Use the
team suggestion form ↓
instead — it adds the internal fields (your team identity, COI considerations, second-opinion routing, fast-tracking) and is the reward-eligible route.
Add more detail (optional) — helps us prioritise faster
Suggestion received — thank you!
Submit your own research
Are you an author — or a research organisation — who would like The Unjournal to commission a public
evaluation of your work? Submit it here for the team to review. Connecting this intake to the team’s main Coda workflow is still in progress.
(To suggest someone else’s research, use the suggestion form above.)
Questions: contact@unjournal.org.
Open the author submission form
Submission received — thank you!
Fields marked * are needed for us to process your submission.
Suggest research — Unjournal team
For Unjournal team members and field specialists. Feeds the same database as the public
suggestion form, but adds the internal fields (your team identity, COI considerations, second-opinion
routing, fast-tracking) and is the reward-eligible route.
Not on the team? Use the
public suggestion form above.
Open the team suggestion form
Prioritisation detail (optional)
Team-only
If the paper fits any of these, tick all that apply:
Suggestion received — thank you!
Fields marked * help us process and credit your suggestion.
Vision: How this tool will work
We are building an efficient, AI-augmented prioritization pipeline:
AI discovery & preliminary rating — The tool finds, vets, and suggests research from multiple sources (NBER, arXiv, SSRN, EA Forum, Anthropic/DeepMind research pages, GovAI/CSET/GPI, etc.), giving a preliminary score and adding it to the prioritization database.
Human suggestions — Team members and the public can also add research directly as a "suggester" or "submitter," in which case the AI provides an additional analysis report.
Notifications — Sign up for alerts when new high-potential research in your area is added.
Team assessment & human labeling — Team members review suggestions, find those of most interest, and give independent ratings. Together with the public quick-rate votes and suggestions collected on this page, these human judgments form a growing set of labeled examples — each one a paper paired with a human verdict on its evaluation priority.
Voting & decisions — The team votes (as in our current process), moving papers forward for commissioned evaluation. The voting outcomes themselves become further labels.
Learning loop — human labels improve the AI — The AI's preliminary scores are systematically compared against these accumulated human labels (team assessments, voting outcomes, and public quick-rate votes), and the model is recalibrated with field-specific corrections — extending the calibration already done against
353 past prioritization decisions.
As more humans rate more papers, the AI's discovery and preliminary scoring (step 1) get steadily more accurate — closing the loop.
The AI uses Unjournal's core principles and previous prioritization decisions as context, and every human rating you contribute — including the quick-rate votes above — feeds back to make its suggestions better.
We welcome your thoughts on this workflow — use the Hypothes.is sidebar or email
contact@unjournal.org.