The ü/ü Conundrum

Original Article ↗ Hacker News Discussion ↗

Normalization vs. collation & the actual bug

  • Many point out the article’s problem is lack of consistent Unicode normalization, not an inherent Unicode flaw.
  • Unicode Normalization Forms (NFC/NFD) handle composed vs. decomposed characters; Canonically equivalent sequences should render identically.
  • Others stress that normalization (NFC/NFD) and collation/search (Unicode Collation Algorithm, TR10) are distinct problems; both matter for equality, sorting, search.

Confusables, security, and search

  • Several discuss Unicode “confusables” (visually similar but distinct code points) for spam, phishing, and abuse.
  • Libraries exist to map strings to regexes or canonical “confusable groups” to catch tricks like ℍ℮1l೦ ≈ “Hello”.
  • There’s skepticism about collapsing everything to a single form, because some characters are legitimately confusable with more than one base letter.

Filesystems, macOS, and normalization pain

  • macOS historically normalized filenames to NFD; APFS changed details but left a messy ecosystem with mixed normalization and tool assumptions.
  • This causes cross-platform bugs (e.g., same-looking filenames colliding, diacritics rendered misplaced, “ß” vs “ss” conflicts).
  • ZFS is praised for configurable normalization; HFS+/APFS and git interactions are cited as concrete failure cases.

Search behavior, diacritics, and locales

  • Strong disagreement on how search “should” treat accents:
    • Some want accent-insensitive search (helping non-native users, keyboard limitations).
    • Others insist that conflating letters (e.g., n vs ñ, cono vs coño) breaks meaning and precise search.
  • Consensus that systems need both exact and fuzzy modes, often locale-aware, and that Unicode alone doesn’t solve language-specific rules (Turkish I, Swedish Å/Ä/Ö, Dutch IJ, Hungarian digraphs, etc.).

Names, passports, and real-world frustration

  • Multiple stories about names with diacritics (e.g., ü, ß, non-Latin scripts) breaking in government, banking, travel, and legacy IT systems.
  • Workarounds include ASCII-izing names, dual official spellings, or even legal name changes; some commenters see this as pragmatic, others as unacceptable.
  • European regulations (e.g., GDPR’s accuracy requirement) are mentioned as external pressure to fix broken name handling.

Unicode design tradeoffs & Han unification

  • One camp criticizes multiple encodings for same-looking characters and combining marks, wishing Unicode had forced one canonical form.
  • Others respond that:
    • Unicode must round-trip legacy encodings and respect distinct scripts and casing behaviors.
    • “Identical-looking glyph → same code point” is ill-defined across fonts, scripts, and history, and Han unification shows how controversial unification already is.
  • Several note that correct handling of text almost always requires locale (and sometimes language tagging), not just raw code points.