Bitten by Unicode

Original Article ↗ Hacker News Discussion ↗

Unicode’s many visually similar hyphens, dashes, and minus signs are causing real-world bugs in code, data pipelines, and scraped documents, especially when text comes from PDFs, Word, or spreadsheets that silently substitute “prettier” characters. Commenters debate how much Unicode should appear in source code, and recommend defenses such as IDE highlighting of confusable characters, strict parsing with validation and errors on unexpected symbols, and normalization or “skeleton” mappings based on Unicode security guidelines. The broader theme is that any system ingesting text must assume messy, inconsistent Unicode and design for robust, explicit handling rather than naive regexes or ASCII-only assumptions.

Dash‑like Unicode characters

  • Many visually similar hyphen/minus/dash characters exist (e.g., HYPHEN-MINUS, HYPHEN, MINUS SIGN, EN/EM dashes, figure dash, hyphen bullet, etc.).
  • Unicode “confusables” tables show numerous mappings between them, often to ASCII -.
  • Some characters (e.g., U+2010 HYPHEN) appear rarely in real documents, raising questions about their practical usefulness.
  • There is disagreement over whether these distinct code points add useful semantics or mostly introduce confusion.

Unicode in source code

  • Some advocate ASCII-only (or highlighting non-ASCII) in code to avoid subtle bugs and copy/paste issues, especially from Word, Outlook, PDFs, LaTeX, etc.
  • Others strongly favor extensive Unicode use in code (identifiers, comments, tests, diagrams, non-English text) for readability and domain fidelity.
  • Linters and syntax highlighters are suggested to differentiate: flag suspicious punctuation outside string literals while allowing rich text inside them.

Normalization, regex, and parsing strategies

  • Suggestions include:
    • Use Unicode properties in regex (\p{Hyphen}, \p{Dash}, categories) where available.
    • Apply Unicode compatibility normalization (NFKC/NFKD) or TR39 “confusables” skeleton mappings, though standard NFC/NFD do not merge hyphen/minus variants.
    • Normalize all hyphen/minus–like characters to ASCII - before further parsing.
  • Others warn that such normalization can erase legitimate semantic differences and that TR39 skeletons were designed for security, not storage/display.

PDFs, spreadsheets, and dirty data

  • PDF text extraction is highlighted as especially error‑prone: fonts map to glyph IDs, and reverse mapping often picks obscure but “closest” Unicode characters.
  • Real‑world datasets (spreadsheets, financial exports, HTML from Word) routinely contain mixed dashes, smart quotes, non‑breaking spaces, odd bullets, and invisible characters.
  • Many argue that robust pipelines inevitably accumulate data‑cleaning rules; some even resort to machine learning for extraction.

Numbers, currency, and error handling

  • Debate over using floats for money: some say floats are common for interim calculations but not settlement; others insist on fixed‑point/decimal or smallest‑unit integers.
  • Locale issues (decimal vs thousands separators, digit sets, grouping conventions) complicate regex‑based numeric parsing.
  • Several recommend strict parsing: match entire strings, reject any unexpected character, and surface errors rather than guessing intent.
  • Others prefer more forgiving normalization of all plausible minus signs, accepting that user intent may trump strict Unicode semantics.