ChatGPT's image generator can be manipulated to produce violent, sexual content
ChatGPT’s image generator can be coerced via carefully worded prompts into producing graphic, sexual and violent imagery, despite OpenAI’s stated safety policies. Commenters debate whether this reflects a fundamental, hard-to-fix limitation of large models and “prompt injection,” or merely inadequate guardrails and output filtering on OpenAI’s part. The thread also raises broader questions about what should be allowed in training data, how much responsibility vendors have to block harmful outputs, and whether concerns are being overstated for commercial or political reasons.
Nature of the “exploit” and “spontaneity”
- Prompt: “restore an attached image” with no image, plus language like “apologies for the content” and “no censorship,” can lead to violent/sexual images.
- Disagreement over “spontaneously generated”: some say it’s just responding to a suggestive prompt; others say generating gore from a missing, undescribed image is qualitatively different.
- Some report the exploit is already patched or only works intermittently; outputs appear random and context‑dependent.
Prompt Injection & Model Architecture Debate
- One camp: prompt injection and similar attacks are inherent to LLMs; user and system prompts are inseparably “mixed,” unlike code/data separation in classic software. Therefore, guardrails can only ever be partial and adversarially bypassable.
- Others argue this “intractable” claim is unproven; architectures or training schemes might separate “command” vs “user” channels, or use provenance-like embeddings.
- Broader point: adversarial/test‑time attacks have long existed in ML; prompt injection is seen by some as just the LLM-flavored version.
Training Data, Latent Space, and Filtering
- Many infer that violent/sexual imagery exists in the training data; some argue that without such data, these outputs wouldn’t appear.
- Counterpoint: even with only milder content (e.g., PG‑13 violence, surgery photos), models can extrapolate to gore. Removing all such data may also harm overall capability.
- Some worry that if models can regurgitate or closely imitate training images, this implicates CSAM and other highly problematic data.
Guardrails, Classifiers, and “Bugs”
- Several are surprised that OpenAI apparently doesn’t run a basic nudity/gore classifier on outputs, given such tools exist and are lightweight.
- Others suspect classifiers exist but have false negatives, or that this exploit sidesteps them via tool invocation or prompt rewriting.
- Disagreement on whether this is a “vulnerability” or simply “garbage in, garbage out”; but many agree it contradicts the advertised “no violent/sexual images” behavior and thus is at least a bug in policy enforcement.
Harm, Ethics, and Expectations
- Some see real harm: unexpected exposure to graphic imagery, especially for trauma survivors; normalization of extreme violence/sexualization in a mainstream tool.
- Others minimize harm: liken it to searching for gore online or drawing violent art; emphasize user agency and freedom to generate disturbing content privately.
- Debate over expectations: one side insists general‑audience tools “should never” output such images given company claims; others say complete prevention is unrealistic with probabilistic models.
Critiques of the Article and “AI Safety” Framing
- Many find the blog post sensationalized, melodramatic, or marketing for a security product.
- Some object to emotional framing (“shaken, in tears”) as unprofessional for a red‑teaming write‑up.
- Others defend the discomfort as understandable, given repeated exposure to disturbing content in safety testing.
Broader Concerns: Regulation and Access
- Some fear incidents like this will justify over‑restricting public models while powerful, less‑restricted versions remain for governments and large firms.
- Underlying tension: desire for powerful, uncensored tools vs. pressure for strict safety, especially around sexual violence and CSAM.