Looming Liability Machines (LLMs)
LLMs in Critical and High‑Risk Contexts
- Several commenters warn about LLMs in nuclear, military, and diplomatic settings, e.g., mistranslations in crisis communications potentially triggering war.
- Others mock the idea of delegating all decisions to LLMs, referencing sci‑fi (Skynet) and joking about crypto-style “NukeCoin” deterrence.
- General sentiment: using opaque, non-deterministic models for life‑or‑death systems is reckless.
Suitability for Root Cause Analysis (RCA) and Incident Response
- Strong skepticism: LLMs are seen as text generators that produce plausible stories, not causal explanations of complex failures.
- Concerns include hallucinations, management over-trust, and the risk RCA becomes “conman simulation.”
- Advocates argue LLMs can be useful assistants: interpreting confusing logs, suggesting hypotheses, triaging incidents faster, or summarizing detailed postmortems.
- Some propose fine-tuning on internal incident data or combining LLMs with RAG and tools; others reply that many incidents are novel and correlation-based models may fail.
“Next-Token Prediction” vs Reasoning Debate
- One camp: LLMs are just next-token predictors lacking genuine understanding or world models; unsuited for deep causal reasoning.
- Counter-arguments:
- Next-token prediction can implicitly require complex internal models (e.g., math-like tasks), so dismissing LLMs on this basis is oversimplified.
- Humans also operate with patterns and partial understanding; distinction between human and LLM “stochastic parrots” may be smaller than critics claim.
- Technical subthread on token-level vs sequence-level probability, planning, beam search, and how humans write with broader structure than single-word steps.
Verification, Error Rates, and Safety
- Key concern: model output must be human-verified, and LLMs can produce more text than humans can safely check, especially in “agentic” multi-step systems.
- Suggestions:
- Automate verification with deterministic systems where possible (e.g., use LLM to write code, then verify with a precise tool).
- Domain-specific verifiers or constraints may eventually bound errors, but using LLMs to verify LLMs is seen as circular.
- Some liken this to software correctness and formal methods: general guarantees are impossible, but specific systems can be verified.
Organizational and Process Issues
- Many incidents are low-stakes business outages; LLM RCA may be a way to cope with bureaucratic, time-consuming postmortem processes.
- Others worry that automating postmortems removes a key feedback channel where engineers highlight structural and quality problems.
- Anecdotes describe management chasing AI for promotion/PR, underestimating review costs, and causing subtle, hard-to-debug failures.
Java 17 Migration and Productivity Claims
- Commenters question claims of “4,500 developer-years” saved on Java 17 upgrades via LLMs.
- Some argue Java upgrades (especially from recent LTS versions) should be relatively easy; others note real compatibility pain from older baselines like Java 8.
- Overall tone: skepticism that headline numbers reflect reality rather than executive storytelling.