Google Gemini tried to kill me

Original Article ↗ Hacker News Discussion ↗

Incident & Food-Safety Context

  • Thread centers on Gemini giving a recipe for cold-infused garlic-in-olive-oil that involves sealing at room temperature for days, which can enable botulism.
  • Several commenters explain that:
    • Botulism risk comes from Clostridium botulinum spores in low-acid, anaerobic environments like oil.
    • Both garlic and oil are safe alone; danger arises when combined and stored improperly.
    • Safe approaches: heating, acidifying (citric/vinegar) plus refrigeration, or short-term use only.
  • Others note botulism is rare relative to other foodborne illness, but consequences are severe.

Responsibility, Intent, and Blame

  • Strong debate on “AI tried to kill me” framing:
    • One side: the model only emits text; responsibility lies with humans (users, engineers, managers, product framing).
    • Other side: giving harmful advice matters regardless of “intent”; if a human said the same thing they might bear some responsibility.
  • Comparisons are made to dogs harming people (headline blames dog, liability on owner) and to GPS sending drivers into rivers.

LLM Reliability, Trust, and Use Cases

  • Consensus: LLMs are useful but fundamentally untrustworthy as authorities, especially for safety-critical domains.
  • Characterizations:
    • “Word/number predictors” optimized to sound right, not be right.
    • “Bullshit machines” whose wrong answers are crafted to look correct.
  • Suggested mental model: treat output like advice from a knowledgeable but sometimes incoherent person; always vet, especially for health/food.

Variability Across Models and Instances

  • Multiple people test the same prompt:
    • ChatGPT and some Gemini runs do warn about botulism and recommend refrigeration.
    • Other Gemini outputs lack warnings, or vary by draft, model version, geography, and safety settings.
  • This non-determinism undermines trust; some see the screenshot as possibly staged but note similar unsafe outputs are reproducible.

Broader Concerns: Deployment, Liability, and Content Pollution

  • Many criticize Google for bolting LLM answers onto search for “true facts” queries.
  • Worries that:
    • Businesses will over-rely on LLMs without human oversight until lawsuits or deaths force change.
    • LLM-generated misinformation will saturate the web, making cross-checking harder.
  • Others argue AI hype overstates productivity gains because all serious uses still require human verification.