The young people sifting through the internet's worst horrors
Commercial content moderation—filtering out child abuse, gore, sexual harassment and other NSFL material from social platforms—is portrayed as psychologically devastating work that is routinely offloaded to young, poorly paid contractors, often in poorer countries. Commenters debate why this labor exists at all, how much harm comes from viewing extreme imagery, whether AI can realistically replace humans without new risks, and to what extent corporations, legal systems and users themselves are complicit in sustaining an exploitative model. Many argue that even as AI helps with detection, people will remain in the loop to handle edge cases and supply training data, meaning the core ethical and mental‑health problems are far from solved.
Psychological impact of moderation work
- Many compare the toll of viewing extreme content (CSAM, gore, beheadings, domestic/sexual abuse) to war, policing, EMS, or traffic police work.
- Repeated exposure is described as causing desensitization, paranoia, PTSD, substance abuse, and “withdrawal” when removed from the stream.
- Others argue not all graphic material is equally harmful: emotional abuse, child sexual assault, and tragic accidents are described as worse than “pure gore.”
- There is disagreement on how inherently damaging images are: some say “what doesn’t kill you makes you stronger,” others strongly reject this as harmful folk wisdom.
Nudity, context, and “dick pics”
- Moderators of dating apps report high volumes of unsolicited genital photos, mostly targeting women.
- Debate centers on whether this is “just nudity” vs. harassment:
- One side emphasizes intent, power, and humiliation; seeing constant targeted abuse leads to disgust and loss of trust in men.
- The other side argues that nudity itself is culturally oversexualized, and that trauma stems from context, not anatomy.
- Volume and repetitiveness are also cited as mentally draining, even when content isn’t extreme.
AI, scale, and edge cases
- Many see AI as essential for detecting NSFL/illegal content and reducing human exposure, but:
- False positives/negatives at scale remain a major risk, especially when automated bans or law-enforcement referrals are tied to AI output.
- Adversaries adapt, so AI is just the first filter; human review remains necessary.
- Training detection models itself requires large human-labeled datasets of horrific material.
- Legal constraints on storing such content (e.g., in some jurisdictions) further complicate AI training.
Corporate structures, outsourcing, and ethics
- Repeated references to content moderation centers (e.g., in Kenya) highlight low pay, poor conditions, union-busting, and legal action against big platforms and their contractors.
- Some argue corporations are structurally profit-maximizing and effectively amoral; others dispute that the law strictly requires shareholder primacy but agree incentives favor ruthless behavior.
- Users’ complicity is noted: people decry exploitation of moderators yet keep using heavily filtered social platforms.
Workforce, training, and transparency
- Commenters argue for better upfront disclosure of job realities, psychological screening, systematic support, and possibly older or more experienced staff.
- Current onboarding is seen as abrupt and maximally shocking, rather than gradually building coping mechanisms.