Why isn't preprint review being adopted?

Limits and Purpose of Peer Review

• Many see peer review as a weak but useful “spam filter” that screens out obvious nonsense, not a guarantee of truth or reproducibility.
• Several argue non-experts overtrust “peer reviewed” labels; within fields, papers are read skeptically regardless of review.
• Others counter that, despite flaws, peer review and journal curation strongly improve the signal‑to‑noise ratio and guide readers toward higher‑value work.

Reproducibility and Replication

• Irreproducibility is often undetectable at publication; non‑replication is framed by some as an expected part of science.
• Others say reviewers can flag likely non‑replicable work using tools like power analysis and by scrutinizing design (e.g., preregistration, controls).
• Major practical obstacles: lack of incentives for replication, high cost (especially in ML and large experiments), and hidden data/methods.
• Suggestions include requiring open data/code, building venues for negative results and replication attempts, and even tying citation rights or publication to independent replication—though critics call this infeasible and biased toward low‑risk work.

Preprints, Preprint Review, and Journals

• In some areas (notably CS/ML), arXiv-style preprints are effectively primary communication; conferences/journals add a “stamp” rather than do core dissemination.
• Open platforms like OpenReview, eLife’s reviewed-preprints, TMLR, PREreview, and git‑based “journals” are discussed as emerging models.
• One camp wants to minimize or abolish pre‑publication peer review, returning to editor-only checks plus open post‑publication critique.
• Others stress that journals still provide crucial curation and reputation signals; without them, valuable work risks being buried in noise.

Incentives, Status, and Workload

• Publish‑or‑perish and prestige metrics (venue, impact) are seen as root causes of salami‑sliced papers, design‑flawed studies, and reluctance to replicate.
• Review is unpaid, time‑consuming expert labor; some propose building reviewer/replicator costs into grants and paying for thorough work.
• Skeptics warn that payment can incentivize low‑effort rubber‑stamping, gaming reviewer ratings, and “consulting‑style” pandering to authors.

Other Concerns and Ideas

• Fears include preprint review becoming another culture‑war battleground or “science by consensus.”
• Proposals range from arXiv comment systems and “Twitter‑style community notes” on papers, to mandatory mini‑reviews when citing, to AI‑assisted code/experiment QA in AI research.
• Several note that public mistrust stems at least as much from sensationalist journalism and institutional incentives as from peer review per se.