Humans missed 1 in 3 threats approving AI agent commands across 40k game runs
An online experiment where players approve or reject AI-generated terminal commands found that humans missed roughly one-third of embedded security threats across 40,000 game runs, even when explicitly warned. Commenters argue this underscores the limits of “human-in-the-loop” permission prompts, which quickly lead to fatigue, rubber‑stamping, and liability offloading rather than real safety. Many instead advocate for stronger technical controls—sandboxing, capability-based security, constrained environments, and AI classifiers that audit other agents—while others note flaws in the game design and stress how hard it is to define what a “secure” AI agent should be in practice.
Effectiveness of Human-in-the-Loop Permissions
- Many see “click to approve” as a failed security pattern repeated from OS permission dialogs and phishing training.
- Users develop “monitor blindness” and click “yes” reflexively, especially under time pressure.
- Some argue 1-in-3 missed threats is catastrophically bad for any serious process; others note the test involves stress, limited context, and non-experts.
- Several point out that in real work, rare threats plus constant prompts almost guarantee eventual failure.
Alternatives: Auto-Mode, Classifiers, Sandboxing
- Popular suggestions: automatic classifiers on each action, separate “auditor” models, plus sandboxing (containers, VMs, microVMs, bytecode-level sandboxes).
- Some already run agents as unprivileged users, inside VMs, with restricted network and filesystem, or products that centrally enforce what agents can talk to.
- Others claim exfiltration is fundamentally hard to prevent if any network access exists; sandbox escapes and supply-chain attacks remain concerns.
What Would a Serious Agent Security Model Be?
- Multiple commenters say it’s unclear how to define a “secure agent” while still allowing useful access to web, files, and tools.
- Simple “allow command X?” prompts are seen as the wrong abstraction; file- and capability-based models, blast-radius containment, and time-based controls are proposed.
- Some liken it to securing a human operator with infinite risk tolerance and no self-preservation.
Capability-Based Security & Languages
- Strong interest in capability-based security, both at OS and language level, to constrain what code (including agent-written code) can ever do.
- Advocates describe capabilities as fine-grained, transitive permissions that could mitigate supply-chain risk and make many libraries inherently non-dangerous.
Liability, UX, and “Moral Crumple Zones”
- Many view permission prompts and warnings as liability shields: shifting blame to users or low-level operators when systems fail.
- Concepts like “moral crumple zones” and “accountability sinks” are invoked to describe humans absorbing blame for complex automated systems.
Critiques of the Game & Data
- Several note the game is timed, has no real consequences, and sometimes labels commands as dangerous in ways they dispute or that depend on missing context.
- Others still see value: even in a toy setting, it highlights how hard and fatiguing command-level review is, and how easily humans miss subtle threats.