You can't just assume UTF-8

Original Article ↗ Hacker News Discussion ↗

Assuming UTF-8 vs Supporting Legacy Encodings

  • Many argue they practically assume or require UTF‑8 for new systems, failing fast on invalid sequences and asking users to fix inputs.
  • Others note this breaks down with legacy data: archives, government/academic datasets, old systems, Japanese and Chinese text (Shift‑JIS, GBK, ARIB encodings), and Windows‑1252 CSVs from Excel are all still common.
  • Several say “assuming UTF‑8 and bailing on errors” is acceptable for greenfield apps, but not when paid to process arbitrary historical files.

Detection, Heuristics, and Their Limits

  • Popular strategy: try UTF‑8 first; if it decodes cleanly, treat it as correct (UTF‑8’s structure makes false positives rare except for pure ASCII).
  • After UTF‑8 fails, some recommend trying a small, known set of legacy encodings; others insist statistics‑based detection is usually more accurate.
  • Multiple comments stress that charset guessers (e.g., naïve byte‑frequency tools) are fragile and often create mojibake that later needs repair (libraries like ftfy exist largely to undo this).
  • Specific encodings like Shift‑JIS, various Windows code pages, and EBCDIC are cited as hard to distinguish reliably; mixed‑encoding files are reported in the wild.

Robustness Principle and Error Handling

  • There’s pushback against Postel’s Law (“be liberal in what you accept”): permissive decoding entrenches malformed data and forces every future implementation to replicate quirks.
  • HTML and browser encoding behavior are cited as cautionary examples: complex specs, auto‑detection, removed “choose encoding” UI, and security concerns.
  • Counterpoint: users blame software that strictly rejects messy real‑world input; products that “just work” with malformed data often win adoption.

Practical Strategies Discussed

  • Separate “cleaning/transcoding” from business logic: one service converts everything to UTF‑8, others assume UTF‑8 only.
  • For CSV and similar: some advocate UTF‑8‑only plus clear errors; others require fallback handling because users export Windows‑1252 or locale‑specific CSVs from Excel and will not configure encodings correctly.
  • On Windows, UTF‑16 APIs and optional UTF‑8 code pages complicate things; BOMs and Unicode normalization (NFC vs NFD, especially on macOS) add another layer of subtle bugs.

Performance and Size Considerations

  • UTF‑16 is denser for pure CJK text, but comments argue that:
    • Real documents mix ASCII markup and CJK, favoring UTF‑8.
    • Compression largely erases size differences.
    • Simplicity and ubiquity of UTF‑8 outweigh small space wins for legacy encodings.