You can't just assume UTF-8

Original Article ↗ Hacker News Discussion ↗

Programmers debate whether it is now reasonable to assume all text is UTF-8, or whether software still needs to detect and handle a long tail of legacy encodings like Windows-1252, Shift-JIS, GBK, and EBCDIC. Many argue for a strict “UTF-8 or error” stance to simplify systems and avoid the long‑term costs of heuristics and Postel-style leniency, while others note real-world cases—CSV exports from Excel, Japanese and Chinese archives, old mainframe data—where guessing or converting encodings is unavoidable. The thread also touches on related pitfalls such as BOMs, Unicode normalization, mojibake, and how over-tolerant parsing (as with HTML and HTTP) has historically led to fragile ecosystems.

Assuming UTF-8 vs Supporting Legacy Encodings

  • Many argue they practically assume or require UTF‑8 for new systems, failing fast on invalid sequences and asking users to fix inputs.
  • Others note this breaks down with legacy data: archives, government/academic datasets, old systems, Japanese and Chinese text (Shift‑JIS, GBK, ARIB encodings), and Windows‑1252 CSVs from Excel are all still common.
  • Several say “assuming UTF‑8 and bailing on errors” is acceptable for greenfield apps, but not when paid to process arbitrary historical files.

Detection, Heuristics, and Their Limits

  • Popular strategy: try UTF‑8 first; if it decodes cleanly, treat it as correct (UTF‑8’s structure makes false positives rare except for pure ASCII).
  • After UTF‑8 fails, some recommend trying a small, known set of legacy encodings; others insist statistics‑based detection is usually more accurate.
  • Multiple comments stress that charset guessers (e.g., naïve byte‑frequency tools) are fragile and often create mojibake that later needs repair (libraries like ftfy exist largely to undo this).
  • Specific encodings like Shift‑JIS, various Windows code pages, and EBCDIC are cited as hard to distinguish reliably; mixed‑encoding files are reported in the wild.

Robustness Principle and Error Handling

  • There’s pushback against Postel’s Law (“be liberal in what you accept”): permissive decoding entrenches malformed data and forces every future implementation to replicate quirks.
  • HTML and browser encoding behavior are cited as cautionary examples: complex specs, auto‑detection, removed “choose encoding” UI, and security concerns.
  • Counterpoint: users blame software that strictly rejects messy real‑world input; products that “just work” with malformed data often win adoption.

Practical Strategies Discussed

  • Separate “cleaning/transcoding” from business logic: one service converts everything to UTF‑8, others assume UTF‑8 only.
  • For CSV and similar: some advocate UTF‑8‑only plus clear errors; others require fallback handling because users export Windows‑1252 or locale‑specific CSVs from Excel and will not configure encodings correctly.
  • On Windows, UTF‑16 APIs and optional UTF‑8 code pages complicate things; BOMs and Unicode normalization (NFC vs NFD, especially on macOS) add another layer of subtle bugs.

Performance and Size Considerations

  • UTF‑16 is denser for pure CJK text, but comments argue that:
    • Real documents mix ASCII markup and CJK, favoring UTF‑8.
    • Compression largely erases size differences.
    • Simplicity and ubiquity of UTF‑8 outweigh small space wins for legacy encodings.