GPT-3.5 crashes when it thinks about useRalativeImagePath too much

Original Article ↗ Hacker News Discussion ↗

A strange bug in GPT‑3.5 causes it to error out or behave incoherently whenever it encounters the rarely used token `useRalativeImagePath`, likely linked to how the tokenizer and training data handle certain “glitch tokens.” Commenters examine possible causes—from malformed embeddings and numerical instability to post‑processing filters and blacklisting—and note that GPT‑4 appears more robust despite sharing the same tokenizer. The incident fuels broader reflections on data quality, adversarial inputs, AI safety “kill words,” and the practical reliability and performance of large language models in real-world applications.

Glitch tokens, tokenization, and useRalativeImagePath

  • Several commenters identify useRalativeImagePath as a known “glitch token” in GPT‑3.5/4’s cl100k_base tokenizer.
  • Hypothesis: such tokens were extremely frequent in the tokenizer’s source corpus (e.g., Reddit usernames, Katalon XML option name with typo), then largely removed before model training, leaving embeddings with no meaningful semantics.
  • Someone links to prior work enumerating glitch tokens and notes the vocabulary is ~100k tokens, not 2^16.
  • Whitespace affects tokenization: the glitch only appears if there is no preceding space before useRalativeImagePath.

Why GPT-3.5 “crashes” and how LLMs generate tokens

  • Multiple commenters emphasize that the model always outputs valid tokens from a fixed vocabulary via logits + softmax.
  • One view: crash-like behavior is probably not “invalid tokens” but downstream issues—e.g., badly conditioned embeddings for these tokens causing numerical instability (NaNs) in inference, or bugs/filters in OpenAI’s serving stack.
  • Others mention special tokens and post-processing layers as potential sources of errors but note that special tokens still originate from the model.
  • Some suggest the token might be blacklisted after earlier “unspeakable token” incidents. This is presented as speculation.

Safety, shibboleths, and “kill words”

  • Several users riff on the idea of deliberately training in “kill words” or “AI safe words” that would shut models down, but others argue this would be an insecure backdoor and anyway operates at tokenizer level, not core “intelligence.”
  • Glitch tokens are compared to shibboleths: phrases humans can easily repeat but that trip up LLMs, potentially useful to distinguish bots from humans.

Training data sources and bias

  • Reddit (including /r/counting) is discussed as likely training data, with estimates of its size and comments that much of online content is repetitive.
  • Some speculate early ChatGPT political bias might stem from heavy Reddit use.

Model behavior, censorship, and reliability

  • Users share anecdotes of GPT‑3.5/4 failing or resetting on certain prompts (including sensitive topics and obscure technical tasks).
  • One commenter notes GPT‑4 also behaves oddly with the glitch token, substituting unrelated words and giving strained explanations.

Meta: UX and performance tangents

  • Several remarks criticize or defend GPT‑4’s API speed and context limits, leading to a heated subthread on token limits and access.
  • Readers complain about the blog’s migraine‑like background; the author later removes it.