A spectre is haunting Unicode
Unicode’s handling of East Asian scripts exposes tensions between abstract character models and messy historical realities. Commenters delve into “ghost characters” like 彁 that entered standards through scanning or encoding errors, the controversial Han unification of Chinese‑Japanese‑Korean ideographs, and the practical fallout for search, fonts, and OCR. While many see these oddities as mostly harmless historical artifacts now that Unicode is no longer constrained to 16 bits, they highlight how hard it is to build a universal encoding for living, visually variable writing systems.
Ghost / “Spectre” Characters
- Several comments find the story of ghost characters (like 彁) fascinating but ultimately low‑stakes: extra codepoints are “harmless historical oddities” now that Unicode is no longer tightly constrained by 16‑bit limits.
- Some see value in keeping incorrect or ghost characters because they enable discussion, historical analysis, and fun uses (e.g., as QA markers, fictional ghost names, tattoos, or a symbol for “unknown/ineffable” concepts).
- Others question why obvious mistakes weren’t simply replaced by the correct characters, viewing strict non-removal as “weird absolutism” that favors error preservation over correctness.
Han Unification and CJK Complexity
- Large subthread debates Han Unification: merging visually or historically related Chinese/Japanese/Korean characters into shared codepoints.
- Critics call the result “an incoherent mess” that breaks expectations for search, font choice, and language identity; they argue it was driven by the need to stay within 16 bits and “fit CJK into the BMP.”
- Others defend the process as the work of serious scholars, note that CJK unification predates Unicode, and stress that Unicode also has many duplicated Latin-like codepoints for compatibility and math semantics.
- There’s disagreement about whether Japanese or Chinese pressure shaped the outcomes; attribution is unclear.
Philosophical / Model Debates
- Some describe Unicode’s character model as “essentialist”: many visually distinct glyphs map to a shared abstract character (e.g., variant “A”s, blackletter, cursive, script differences between regions).
- Others point out that Unicode does not unify everything (Latin/Greek/Cyrillic A, or many accented forms), so Han Unification feels uniquely inconsistent to CJK users.
- Discussion touches on philosophers critiquing Unicode’s approach to “character vs glyph,” but details are mostly second-hand and remain unclear.
OCR, Origins, and Evidence
- Multiple comments discuss the likely origin of 彁 as a misreading of 彊 in a poorly scanned newspaper; some sources in Japanese are mentioned.
- Debate over OCR: older OCR for CJK was slow and unreliable; training on rare or nonexistent characters is hard.
- There’s some technical speculation about OCR using radicals, but precise historical reconstruction is acknowledged as uncertain.
Miscellaneous Encoding Notes
- Side discussions cover:
- Font-dependent differences (Chinese vs Japanese glyphs, Bulgarian vs Russian Cyrillic).
- Compose-key usage for rare symbols.
- Examples of odd Latin letters (ÿ, Ÿ) and combining marks vs precomposed characters.