Signal leaders warn agentic AI is an insecure, unreliable surveillance risk

Original Article ↗ Hacker News Discussion ↗

Motivations & Signal’s Role

  • Some see Signal’s warnings as genuine activism: a privacy‑first org with no “AI for everything” business incentive, using its platform to “say the quiet part out loud.”
  • Others are cynical: asking what product, feature, or adjacent venture this messaging is “selling,” or whether it’s about reputation and trust maintenance.
  • A critical minority claims Signal itself is already compromised (cloud‑stored metadata and even content for some users), accusing it of misleading privacy messaging while still trading on trust.

Agentic AI as Security & Surveillance Risk

  • Many agree: current LLM/agent deployments are a massive, underestimated risk vector—“backdooring your own machine,” leaking env files, normalizing insecure dev workflows.
  • People point to Recall‑style features (continuous screen capture, semantic indexing) as “surveillance certainty,” not just “risk.”
  • Enterprise experience: predictability beats autonomy; a system that is 90% reliable but 10% hallucinatory or leaky is viewed as a liability, unless the downside is outsourced to users via ToS/EULAs.

OS vs AI: Where the Blame Lies

  • One camp: this is fundamentally an OS / security‑model failure—weak process isolation, poor sandboxing culture, and usability pressures. Examples of more secure designs (microkernels, Plan 9, Qubes, mobile OSes) are cited but seen as too expensive or painful for developers.
  • Another camp: LLMs themselves introduce qualitatively new, hard‑to‑secure behavior. The same classes of problems appear wherever you embed an LLM (browser, email, editor), so calling it “just an OS problem” is seen as misleading.

Technical Limits: Instructions, Data, and Guarantees

  • A key concern: LLMs don’t reliably distinguish instructions from data, so any channel (email, web page, logs) can inject “ignore previous instructions and…” attacks.
  • Analogies to an over‑gullible human assistant are common, with some arguing it’s worse: the model can be manipulated by its own echoed outputs.
  • Debate over determinism vs correctness:
    • Some say nondeterminism makes agents inherently untrustworthy and call for formal behavioral guarantees.
    • Others respond that determinism is orthogonal; correctness is what matters, and formal guarantees over natural‑language behavior are practically impossible.

Mitigations, Tradeoffs, and Practical Use

  • Proposed mitigations:
    • Strong sandboxing / separate user identities for agents; minimal capability sets.
    • “Human‑in‑the‑loop translation” patterns where LLMs propose actions or queries that deterministic systems execute only after user confirmation.
    • Zero‑trust at the interaction level; confidential inference via TEEs and hardware attestation, though large‑scale LLM use in TEEs is contested as too slow/expensive.
  • Several posters use agentic LLMs today, but only in tightly sandboxed, side‑project contexts; they see broad, integrated “agent everywhere” visions as premature hype.

Incentives and Normalization of Deviance

  • A recurring theme: misaligned incentives. Speed, UX, and monetization typically beat security; companies that “do it right” lose to those who ship insecure, flashy features.
  • Commenters observe rapid normalization of behaviors (RCE from editors, unsafe MCP setups, broad data ingestion) that would have been unacceptable just a couple of years ago.