The ü/ü Conundrum

Original Article ↗ Hacker News Discussion ↗

Unicode’s multiple ways of encoding visually identical characters — such as ü as a single code point or as “u” plus a combining diaeresis — cause subtle bugs in search, file handling, and identity data. Commenters explore normalization (NFC/NFD), collation, and “confusables,” noting how operating systems, filesystems, and programming languages differ in their handling and how this impacts everything from spam filtering to people’s legal names. Many argue that consistent normalization and locale‑aware comparison are essential, but also highlight that these problems reflect the deeper complexity and politics of representing real-world language in software.

Normalization vs. collation & the actual bug

  • Many point out the article’s problem is lack of consistent Unicode normalization, not an inherent Unicode flaw.
  • Unicode Normalization Forms (NFC/NFD) handle composed vs. decomposed characters; Canonically equivalent sequences should render identically.
  • Others stress that normalization (NFC/NFD) and collation/search (Unicode Collation Algorithm, TR10) are distinct problems; both matter for equality, sorting, search.

Confusables, security, and search

  • Several discuss Unicode “confusables” (visually similar but distinct code points) for spam, phishing, and abuse.
  • Libraries exist to map strings to regexes or canonical “confusable groups” to catch tricks like ℍ℮1l೦ ≈ “Hello”.
  • There’s skepticism about collapsing everything to a single form, because some characters are legitimately confusable with more than one base letter.

Filesystems, macOS, and normalization pain

  • macOS historically normalized filenames to NFD; APFS changed details but left a messy ecosystem with mixed normalization and tool assumptions.
  • This causes cross-platform bugs (e.g., same-looking filenames colliding, diacritics rendered misplaced, “ß” vs “ss” conflicts).
  • ZFS is praised for configurable normalization; HFS+/APFS and git interactions are cited as concrete failure cases.

Search behavior, diacritics, and locales

  • Strong disagreement on how search “should” treat accents:
    • Some want accent-insensitive search (helping non-native users, keyboard limitations).
    • Others insist that conflating letters (e.g., n vs ñ, cono vs coño) breaks meaning and precise search.
  • Consensus that systems need both exact and fuzzy modes, often locale-aware, and that Unicode alone doesn’t solve language-specific rules (Turkish I, Swedish Å/Ä/Ö, Dutch IJ, Hungarian digraphs, etc.).

Names, passports, and real-world frustration

  • Multiple stories about names with diacritics (e.g., ü, ß, non-Latin scripts) breaking in government, banking, travel, and legacy IT systems.
  • Workarounds include ASCII-izing names, dual official spellings, or even legal name changes; some commenters see this as pragmatic, others as unacceptable.
  • European regulations (e.g., GDPR’s accuracy requirement) are mentioned as external pressure to fix broken name handling.

Unicode design tradeoffs & Han unification

  • One camp criticizes multiple encodings for same-looking characters and combining marks, wishing Unicode had forced one canonical form.
  • Others respond that:
    • Unicode must round-trip legacy encodings and respect distinct scripts and casing behaviors.
    • “Identical-looking glyph → same code point” is ill-defined across fonts, scripts, and history, and Han unification shows how controversial unification already is.
  • Several note that correct handling of text almost always requires locale (and sometimes language tagging), not just raw code points.