Hacker plants false memories in ChatGPT to steal user data in perpetuity

Original Article ↗ Hacker News Discussion ↗

Local vs Cloud LLMs

  • Many advocate running models locally for privacy, control, and “uncensored” use cases.
  • Recommended stacks: Llama 3.1 (esp. 8B), InternLM2 7B, Gemma/Gemma 2, Codestral, Aya, via tools like Ollama, GPT4All, LM Studio, Msty, OpenWebUI.
  • Hardware views:
    • For 20–70B models with Q4 quantization, high‑VRAM GPUs (e.g., 16–24 GB) are preferred.
    • Smaller (≤13B) models can run on CPU with enough RAM but are slower.
    • Quantization quality differs: Q4 often degrades output noticeably vs Q8/fp16.
  • Consensus: small local models are decent for translation, summaries, and as memory aids, but unreliable for precise or obscure topics.

Prompt Injection, Memory, and Data Exfiltration

  • Several point out that running locally does not inherently fix prompt injection or exfiltration: if the model reads untrusted content, it can be “hacked” regardless of location.
  • Core limitation: LLMs do not reliably distinguish “instructions” from “data” or separate channels (user text, web pages, their own past output).
  • The discussed attack uses hidden instructions (e.g., in an image or document) to:
    • Force the model to call URLs that encode user inputs/outputs.
    • Store this behavior in persistent memory so future chats keep exfiltrating.
  • Some compare this to remote code execution / running a random executable from the internet; others stress that enabling memory by default makes it a notable new vector rather than user misconfiguration.

Mitigations and Red‑Team Dynamics

  • Ideas: a separate model to detect/flag injection attempts; tools like Llama Guard; stronger observability; treating all prompts as untrusted input that must be sanitized.
  • Multiple comments expect an “arms race” where attacking remains cheaper than defending; prior machine‑learning security experience suggests defense is harder and more expensive.

Usefulness vs. Risk of LLMs

  • Strong divide:
    • Enthusiasts report major productivity/learning boosts: rapid prototyping, code scaffolding and unit tests, complex text editing, finding libraries/tools, legal issue overviews, creative writing, studying complex topics interactively.
    • Skeptics highlight frequent hallucinations, misleading confidence, and user over‑trust, especially for legal/financial advice and technical answers that reference non‑existent APIs or cases.
  • Some call LLMs “idiot amplifiers” that let non‑experts produce authoritative‑looking nonsense, which then gets repeated, upvoted, and eventually scraped back into future training data.
  • Ongoing debate: whether this is mainly a user‑education problem or a design/marketing failure, given LLMs are presented as general‑purpose “AI” rather than as fallible text generators.