PEP 686 – Make UTF-8 mode default

Original Article ↗ Hacker News Discussion ↗

Python defaults and PEP 686

  • Many assumed Python 3 already “just used UTF‑8 everywhere”; discussion clarifies that:
    • bytes.decode / str.encode have defaulted to UTF‑8 since 3.2.
    • open() used the locale-dependent encoding; sys.getfilesystemencoding() and locale.getpreferredencoding() can differ.
  • PEP 686 making UTF‑8 mode the default is widely seen as a welcome “repair” to Python 3, especially to avoid platform‑dependent bugs (notably on Windows).
  • Several argue developers should not rely on OS locales at all; non‑UTF‑8 encodings today are usually accidental and should be explicitly requested when intentional.

Platform and locale headaches

  • Windows:
    • Historically uses “ANSI” code pages; UTF‑8 is only opt‑in (global setting or manifest activeCodePage).
    • Changing the whole OS to UTF‑8 by default is seen as politically and technically too disruptive; lots of legacy software assumes old code pages or even ASCII.
    • Examples: PowerShell historically emitting UTF‑16; tools choking on UTF‑16 .gitignore or CSV files.
  • Unix/Linux:
    • Defaults historically varied by locale (e.g., ISO‑8859‑1/‑15); system scripts sometimes still run with non‑UTF‑8 locales, exposing bugs.
    • Some software still mishandles multibyte UTF‑8 or non‑BMP code points.

BOM, utf-8-sig, and interoperability

  • Many Linux tools (including Python when using plain utf‑8) choke on UTF‑8 with BOM, even though it’s technically allowed and common from Windows tools.
  • Python’s utf-8-sig codec exists to handle optional BOMs, but is obscure in docs.
  • Debate:
    • One side wants BOMs “handled gracefully” by default on input.
    • Others resist: a BOM is also a real character (U+FEFF), and automatic stripping breaks exact round‑trip of text; also UTF‑8 BOMs are now generally discouraged.

String representation and Unicode subtleties

  • Clarifications:
    • Python str is a sequence of Unicode code points (not “UTF‑8 strings”); internal storage uses 1/2/4‑byte arrays depending on max code point, with cached UTF‑8 when needed.
    • This allows O(1) indexing but permits “ill‑formed” strings containing surrogate code points that cannot be encoded in standard UTFs.
  • Some consider this design flawed; others note that correct user‑visible “characters” are grapheme clusters anyway, so code‑point indexing is already a compromise.

Broader reflections

  • Several commenters view global locale defaults (for both numbers and text) as a long‑term design mistake.
  • Line endings (CRLF vs LF) and encodings are cited together as persistent cross‑platform friction that modern tools and languages are slowly, but not completely, ironing out.