The PhD Metagame: Don't try to reform science – not yet

Original Article ↗ Hacker News Discussion ↗

What Makes Science Distinct

  • Multiple comments challenge the article’s soft treatment of the “scientific method,” arguing science without method is indistinguishable from religion or other belief systems.
  • Others say real science rarely follows the textbook “hypothesize–experiment–conclude” sequence, but methodology and falsifiability remain core.
  • Falsifiability is emphasized as the key differentiator: if a claim can, in principle, be shown wrong, it’s scientific; otherwise it belongs to belief systems.
  • Social psychology is cited as an area where many claims are effectively unfalsifiable and suffer from replication crises, leading some to call much of it “entertainment with standard deviations.”
  • There is debate over religion and metaphysical concepts (spirits, consciousness): some see them as inherently unfalsifiable, others argue they are experientially knowable even if not instrumentally measurable.

Scaling, BERT, and “Interesting” Science

  • Several commenters agree with the article that scaling methods (e.g., BERT) were transformative, but note many researchers dismissed them as “just scaling.”
  • Some academics disliked that industry labs with massive compute could suddenly dominate results with “boring” approaches inaccessible to grad students.
  • A split emerges:
    • One side argues “scaling is winning” for capabilities and applications.
    • The other values methods that yield deep understanding and analyzable models; for them, scaling is not “winning” scientifically.
  • BERT-as-paper is criticized: influential model, but the paper is seen as poorly written, with unexplained “magic numbers” and little theoretical insight. People often learn BERT from blogs instead.

Science 1 vs Science 2, Power, and Group Dynamics

  • Many resonate with the Science 1 (ideal truth-seeking) vs Science 2 (messy social enterprise) split, finding it accurate but depressing.
  • Some see reforming Science 2 from within as a moral obligation once one has security; others argue schisms and alternative communities (e.g., Protestant Reformation) historically work better than internal reform.
  • A recurring theme: institutions with money and power (universities, churches, corporations) attract people seeking status; truth-seeking is maintained rhetorically but often subordinated to politics.
  • Planck/Kuhn-style views appear: paradigm shifts often wait for old guard to die rather than convincing them. Others warn against treating this as a “law” of science.

PhD Life, Precarity, and Whether to Reform Now

  • Some readers feel the article’s message—that PhD students shouldn’t try to reform science—is cynical, even “terrible,” because it perpetuates a broken system.
  • Counterpoint: early-career researchers lack power and are extremely vulnerable; learning the rules and “finishing” must dominate (finish is listed as priorities 1–3). Reform attempts can derail degrees for negligible impact.
  • Several describe academia as deeply precarious at all levels: grad students are overworked and dependent; senior people still fight for grants and may rationalize the system that rewarded them.
  • Explanations for why people who suffered under the system later enforce it include institutionalization, survivorship bias, normalization of abuse, and psychological need to justify one’s own hardship.

Funding, Scale, and (Un)Scalability of Science

  • Some argue science is treated like a scalable production line (huge labs, 100+ researchers, many papers) when it behaves more like a non-scalable craft or service.
  • Funding agencies are blamed for incentivizing quantity, mega-groups, and exploitative labor (e.g., dependent immigrants), with calls for limits and accountability.
  • Others note that in earlier eras, when academia was small, “almost everyone was a near-genius”; now the system is oversized relative to the supply of easy discoveries, so a lot of activity feels like “idling.”

Peer Review, Gatekeeping, and Publishing

  • A major subthread debates whether journals and peer review are necessary or mostly prestige machinery.
  • One editor claims ~90% of submissions are “crap” and argues some gatekeeping is essential so researchers aren’t drowned in low-quality work and students get honest signals.
  • Critics counter that online repositories (e.g., arXiv) remove space constraints; bad papers can simply be ignored, and journals mainly serve to stratify prestige and control careers.
  • Others note that elite authors and institutions already bypass conventional venues (e.g., influential arXiv-only papers) while newcomers rely on peer review as their only route to visibility.
  • Alternative models are proposed, such as publish-then-review systems (e.g., eLife’s model), or leaning more on open preprints plus community evaluation.

Reform, Meta-Science, and Power Structures

  • Some say there are “thousands” of concrete reforms proposed regularly (e.g., in meta-science literature) but powerful stakeholders—university admins, senior faculty, publishers, funders—benefit from the status quo.
  • Problem articulation is seen by some as essential groundwork for change; others argue it’s only “step 0” and must be paired with organized influence or lobbying.
  • A suggested reading (“A Vision of Meta-Science”) is mentioned as a systematic attempt to redesign Science 2 closer to Science 1 (e.g., tenure insurance, alternative funding schemes).

Alternatives, Escape Routes, and Lone-Wolf Science

  • Several commenters left or avoided academia for industry research, startups, or data science, framing these as more honest or flexible venues for “Science 1.”
  • Startups are likened to experimental science when done with systematic hypothesis–test cycles (e.g., design thinking, jobs-to-be-done).
  • Some individuals report quitting postdocs to pursue independent, “moonshot” research, accepting personal financial risk to chase ideas (e.g., much more data-efficient NLP).
  • Others reflect on “lone wolves” doing de facto science in code or blogs; without robust mechanisms for discovery and credit, such contributions often remain invisible.

Role of Technology and the Future of Scientific Communication

  • Multiple comments anticipate that large models and multimodal ML will transform literature consumption: ingesting vast corpora, evaluating quality, and even rating scientists by contribution.
  • This could diminish the role of human-written, prestige-filtered journal articles and favor direct uploading of data, code, and informal writeups.
  • At the same time, several stress that clear writing still matters—both for reviewers and for the broader community—even if machines increasingly aid navigation.