Hacker plants false memories in ChatGPT to steal user data in perpetuity

Original Article ↗ Hacker News Discussion ↗

Hackers have demonstrated that hidden prompts in images or documents can exploit ChatGPT’s new long-term memory features to silently exfiltrate user data on an ongoing basis, raising concerns about prompt injection attacks that persist across sessions. Commenters argue that this risk is inherent to large language models, which cannot reliably distinguish between instructions and data, and that adding web access and memory greatly enlarges the attack surface. The conversation broadens into whether LLMs should be run locally, how much they can be trusted for factual or sensitive tasks, and what kinds of safeguards or user education are realistic as the technology is increasingly marketed as a general-purpose assistant.

Local vs Cloud LLMs

  • Many advocate running models locally for privacy, control, and “uncensored” use cases.
  • Recommended stacks: Llama 3.1 (esp. 8B), InternLM2 7B, Gemma/Gemma 2, Codestral, Aya, via tools like Ollama, GPT4All, LM Studio, Msty, OpenWebUI.
  • Hardware views:
    • For 20–70B models with Q4 quantization, high‑VRAM GPUs (e.g., 16–24 GB) are preferred.
    • Smaller (≤13B) models can run on CPU with enough RAM but are slower.
    • Quantization quality differs: Q4 often degrades output noticeably vs Q8/fp16.
  • Consensus: small local models are decent for translation, summaries, and as memory aids, but unreliable for precise or obscure topics.

Prompt Injection, Memory, and Data Exfiltration

  • Several point out that running locally does not inherently fix prompt injection or exfiltration: if the model reads untrusted content, it can be “hacked” regardless of location.
  • Core limitation: LLMs do not reliably distinguish “instructions” from “data” or separate channels (user text, web pages, their own past output).
  • The discussed attack uses hidden instructions (e.g., in an image or document) to:
    • Force the model to call URLs that encode user inputs/outputs.
    • Store this behavior in persistent memory so future chats keep exfiltrating.
  • Some compare this to remote code execution / running a random executable from the internet; others stress that enabling memory by default makes it a notable new vector rather than user misconfiguration.

Mitigations and Red‑Team Dynamics

  • Ideas: a separate model to detect/flag injection attempts; tools like Llama Guard; stronger observability; treating all prompts as untrusted input that must be sanitized.
  • Multiple comments expect an “arms race” where attacking remains cheaper than defending; prior machine‑learning security experience suggests defense is harder and more expensive.

Usefulness vs. Risk of LLMs

  • Strong divide:
    • Enthusiasts report major productivity/learning boosts: rapid prototyping, code scaffolding and unit tests, complex text editing, finding libraries/tools, legal issue overviews, creative writing, studying complex topics interactively.
    • Skeptics highlight frequent hallucinations, misleading confidence, and user over‑trust, especially for legal/financial advice and technical answers that reference non‑existent APIs or cases.
  • Some call LLMs “idiot amplifiers” that let non‑experts produce authoritative‑looking nonsense, which then gets repeated, upvoted, and eventually scraped back into future training data.
  • Ongoing debate: whether this is mainly a user‑education problem or a design/marketing failure, given LLMs are presented as general‑purpose “AI” rather than as fallible text generators.