Scientists should use AI as a tool, not an oracle
AI language models are praised as powerful tools for brainstorming, coding and sifting through large bodies of information, but many commenters warn they are often treated as infallible oracles despite frequent hallucinations and fabricated sources. Participants argue that both scientists and the general public need better understanding of how these systems work, their limits, and how to validate their output, especially as they are integrated into research, healthcare, and public-sector decision-making. Underlying the debate are concerns about misplaced trust in machines over human experts, the hype-driven misuse of statistical models, and the risk that the “AI” label obscures corporate responsibility for failures.
AI as Tool vs Oracle
- Strong agreement that AI (especially LLMs) should be a tool, not an authority.
- Concern that many users, including some scientists, effectively let AI “write for them” or source facts without verification.
- Some suggest formal norms like citing AI as a writing assistant to increase transparency.
Hallucinations, References, and Reliability
- Multiple reports of fabricated or misrepresented citations; one user counted ~95% fake references.
- Complaints that LLMs distort even simple factual text when asked to “rewrite” in different tones.
- Clarification that hallucination is not a “bug” in the code path but an inherent consequence of the modeling approach.
Comparison to Search and Source Evaluation
- LLMs differ from search because they strip away context, provenance, and competing answers.
- Traditional search lets users judge credibility via site, author, and links; LLMs offer a single, confident narrative.
- Some think people overestimate their ability to detect unreliable web data anyway.
Use in Science, Academia, and Public Sector
- Worry that scientists will use chatbots to interpret results or draft papers, driven by publish‑or‑perish pressures.
- A contrasting view from public‑sector science: AI assistants could help triage huge backlogs (e.g., toxicology literature, QSAR trends) if used under expert oversight.
- Serious concern about public bureaucracies replacing human checks with AI for efficiency, leading to harmful decisions.
Expertise, Trust, and Epistemology
- Broader problem: people confidently argue against domain experts while uncritically trusting machines.
- Counterpoint: some “experts” in high‑profile domains are politicized, so laypeople struggle to know whom to trust.
- Several note that more information of lower average quality worsens existing epistemic problems.
Definitions: Leakage, Overfitting, and “Curve Fitting”
- “Leakage” discussed as using information in training that would not be available at inference, often via mislabeled or improperly split data; related to but distinct from overfitting.
- Example: models learning background artifacts (e.g., trees) instead of the intended object.
- Some argue calling it “curve fitting” rather than “AI” would demystify it and clarify legal responsibility.
Intelligence, Correctness, and Anthropomorphism
- Debate over whether LLMs are “intelligent” or just probabilistic text generators.
- One side stresses they merely predict next tokens and lack concepts like truth or correctness internally.
- Others insist correctness is still a meaningful external criterion: if the output fails the user’s task, it is wrong, regardless of internal mechanics.
- Warnings that anthropomorphizing models and marketing them as oracles causes misuse and misplaced trust.
Safety, Influence, and Corporate Incentives
- Speculation about using AI to subtly steer human behavior (e.g., toward “better” choices), with ethical and trust risks.
- View that corporations will integrate imperfect AI wherever it is economically beneficial, but must surround it with validation pipelines, as in software development.
- Some skepticism toward highly speculative AI‑doomer narratives; emphasis that genuine safety work should be grounded in real technical understanding.