Human study on AI spear phishing campaigns
Language models are making targeted phishing emails dramatically cheaper and more convincing, raising concern that sophisticated “spear phishing” will be effectively commoditized. Commenters highlight how current defenses still rely heavily on individual vigilance, while corporate practices, confusing verification flows, HTML email, and link rewriting often mimic scam patterns and undermine user trust. Many expect an arms race in which AI systems both generate and filter malicious messages, and argue that deeper protocol, UX, and regulatory changes are needed beyond training people not to click links.
Perceived severity and human vulnerability
- Commenters report frequent, often crude “CEO” and invoice scams via email and SMS, and note that non‑technical and technical people alike fall for them.
- Many argue that anyone can be socially engineered in the right context; “grandma” is no longer a special case.
- Some see the everyday inbox as an active “warzone” where a small mistake can cause major harm.
AI’s role in spear phishing
- Many view AI‑generated spear phishing as a terrifying but likely already‑active threat: it automates personalization and drastically lowers cost.
- Others note this commoditization might, paradoxically, reduce long‑term risk by flooding channels with so much junk that they become unusable or force systemic changes.
- There is speculation about an “agent vs agent” future: attackers’ AI vs defenders’ AI triaging email and information.
Study design, metrics, and ethics
- The study used data scraped from public profiles to personalize messages and framed them as “targeted marketing emails,” with IRB approval.
- Several think defining “success” as merely clicking a link is weak; security‑savvy users may click in sandboxes or out of curiosity without being truly compromised.
- Some worry about bias if participants knew they were in a study; others question whether any non‑consensual design would be ethical.
Email, OS security, and structural issues
- Strong frustration that email security still relies on individuals not clicking links.
- Critiques of deny‑list models (spam filters) vs trust/allow‑list models, though some argue strict whitelisting would kill the open internet.
- Debates about desktop sandboxing, running as admin, and whether operating systems and email clients should better isolate risky actions.
- Divided views on HTML email: some see it as unnecessary attack surface driven by marketing; others note plain text cannot prevent social engineering.
Legitimate services mimicking scams (“scamicry”)
- Numerous anecdotes where banks, big tech, ecommerce sites, and even governments send messages that look indistinguishable from phishing: odd domains, third‑party verifiers, cryptic links, SMS fines, and callback flows.
- This erodes trust and blurs the line between legitimate and malicious contact; users end up ignoring or auto‑filtering even important internal or security emails.
Ideas for defenses and future directions
- Suggestions include stronger trust networks, email aliases, removing HTML, clearer “we will only contact you from X addresses” policies, and more realistic internal phishing tests (measuring more than clicks).
- Some expect social media exposure and OSINT to keep raising risk and foresee growing reliance on AI and other tools just to assess what is real online.