Peer review is an honor-based system (2008)

Original Article ↗ Hacker News Discussion ↗

Peer review in science is portrayed as a low-bar, honor- and reputation-based filter that provides a basic sanity check rather than a guarantee of truth. Commenters highlight systemic problems — bias, gatekeeping, corrupt incentives, poor reviewer accountability, and the misuse of “peer-reviewed” status in policy, media, and career evaluation — while noting that flawed work and even fraud routinely pass through. Alternatives and reforms such as open peer review, preprint servers, and more transparent, collaborative evaluation are suggested as ways to improve signal-to-noise without overburdening researchers or entrenching publishers’ power.

Role and Limits of Peer Review

  • Widely framed as a low bar / sanity check, not a guarantee of truth or scientific consensus.
  • A paper’s real value is seen as its later influence and replication, not its acceptance.
  • Some argue that if papers are “just conversation starters,” they should not be overused for rewards or policy; others respond that professionals already interpret them that way.
  • Concern that the public and journalists treat “peer reviewed” as “proven fact,” fueling both hype and backlash.

Replication, Timescales, and Impact

  • Several comments stress that validation comes from reproduction over time, not initial review.
  • Distinction drawn between “seminal” papers (heavily scrutinized, replicated) and “filler” papers (incremental work and CV material).
  • Historical examples (ulcer bacteria, gut biome, Alzheimer’s debates) used to show how consensus can resist new evidence for years.

Failures, Biases, and Incentives

  • Peer review often misses fraud and fabrication; high-profile scandals and hoax papers are cited.
  • Reports of reviewers blocking or slowing competing work, demanding self-citations, or enforcing narrow fashions; seen more severe in some journals than conferences, or vice versa, depending on field.
  • Conference-centric systems (esp. in CS/ML) criticized as adversarial, prestige-driven, and vulnerable to low-effort or bad-faith reviews.
  • Grant peer review and publish-or-perish metrics are seen as steering research away from “blue sky” work and toward safe, fundable topics.

Alternative and Evolving Models

  • Strong support for public and ongoing peer review: open reports, visible discussion, reputation signals for reviewers.
  • Examples mentioned of journals and conferences using open or signed reviews and arXiv-style preprints with community commentary.
  • Suggestions include separate venues for tentative vs. solidly replicated results, and new “stamps of approval” beyond citation counts.

Use Outside the Scientific Community

  • Biggest dysfunction noted when policymakers, funders, and hiring committees use peer review as a cheap proxy for deep evaluation.
  • Some see this as a journalism/communication failure; others as a structural misuse of a tool designed for expert-to-expert filtering.

Technology and AI

  • Concern that generative AI is being used to write reviews; calls for accountability and signed reviews.
  • Counterpoint: tools are acceptable if humans remain responsible for correctness, though there’s unease about an ecosystem of AI-written papers reviewed by AI.