Looming Liability Machines (LLMs)

Original Article ↗ Hacker News Discussion ↗

LLMs in Critical and High‑Risk Contexts

  • Several commenters warn about LLMs in nuclear, military, and diplomatic settings, e.g., mistranslations in crisis communications potentially triggering war.
  • Others mock the idea of delegating all decisions to LLMs, referencing sci‑fi (Skynet) and joking about crypto-style “NukeCoin” deterrence.
  • General sentiment: using opaque, non-deterministic models for life‑or‑death systems is reckless.

Suitability for Root Cause Analysis (RCA) and Incident Response

  • Strong skepticism: LLMs are seen as text generators that produce plausible stories, not causal explanations of complex failures.
  • Concerns include hallucinations, management over-trust, and the risk RCA becomes “conman simulation.”
  • Advocates argue LLMs can be useful assistants: interpreting confusing logs, suggesting hypotheses, triaging incidents faster, or summarizing detailed postmortems.
  • Some propose fine-tuning on internal incident data or combining LLMs with RAG and tools; others reply that many incidents are novel and correlation-based models may fail.

“Next-Token Prediction” vs Reasoning Debate

  • One camp: LLMs are just next-token predictors lacking genuine understanding or world models; unsuited for deep causal reasoning.
  • Counter-arguments:
    • Next-token prediction can implicitly require complex internal models (e.g., math-like tasks), so dismissing LLMs on this basis is oversimplified.
    • Humans also operate with patterns and partial understanding; distinction between human and LLM “stochastic parrots” may be smaller than critics claim.
  • Technical subthread on token-level vs sequence-level probability, planning, beam search, and how humans write with broader structure than single-word steps.

Verification, Error Rates, and Safety

  • Key concern: model output must be human-verified, and LLMs can produce more text than humans can safely check, especially in “agentic” multi-step systems.
  • Suggestions:
    • Automate verification with deterministic systems where possible (e.g., use LLM to write code, then verify with a precise tool).
    • Domain-specific verifiers or constraints may eventually bound errors, but using LLMs to verify LLMs is seen as circular.
  • Some liken this to software correctness and formal methods: general guarantees are impossible, but specific systems can be verified.

Organizational and Process Issues

  • Many incidents are low-stakes business outages; LLM RCA may be a way to cope with bureaucratic, time-consuming postmortem processes.
  • Others worry that automating postmortems removes a key feedback channel where engineers highlight structural and quality problems.
  • Anecdotes describe management chasing AI for promotion/PR, underestimating review costs, and causing subtle, hard-to-debug failures.

Java 17 Migration and Productivity Claims

  • Commenters question claims of “4,500 developer-years” saved on Java 17 upgrades via LLMs.
  • Some argue Java upgrades (especially from recent LTS versions) should be relatively easy; others note real compatibility pain from older baselines like Java 8.
  • Overall tone: skepticism that headline numbers reflect reality rather than executive storytelling.