PEP 686 – Make UTF-8 mode default

Original Article ↗ Hacker News Discussion ↗

Python’s move to make UTF‑8 the default text encoding (PEP 686) is broadly welcomed as a way to reduce subtle, platform-dependent bugs caused by system locale and legacy code pages, especially on Windows. Commenters contrast external file encodings with internal string representations in Python, Java, and other languages, and recount how earlier defaults (UTF‑16, ISO‑8859 variants, CP1251, UTF‑16 from PowerShell, BOM-heavy workflows) have long caused interoperability problems. Many see explicit, UTF‑8‑centric handling of text as the modern baseline, with non‑UTF‑8 encodings now viewed as edge cases that should be opted into deliberately.

Python defaults and PEP 686

  • Many assumed Python 3 already “just used UTF‑8 everywhere”; discussion clarifies that:
    • bytes.decode / str.encode have defaulted to UTF‑8 since 3.2.
    • open() used the locale-dependent encoding; sys.getfilesystemencoding() and locale.getpreferredencoding() can differ.
  • PEP 686 making UTF‑8 mode the default is widely seen as a welcome “repair” to Python 3, especially to avoid platform‑dependent bugs (notably on Windows).
  • Several argue developers should not rely on OS locales at all; non‑UTF‑8 encodings today are usually accidental and should be explicitly requested when intentional.

Platform and locale headaches

  • Windows:
    • Historically uses “ANSI” code pages; UTF‑8 is only opt‑in (global setting or manifest activeCodePage).
    • Changing the whole OS to UTF‑8 by default is seen as politically and technically too disruptive; lots of legacy software assumes old code pages or even ASCII.
    • Examples: PowerShell historically emitting UTF‑16; tools choking on UTF‑16 .gitignore or CSV files.
  • Unix/Linux:
    • Defaults historically varied by locale (e.g., ISO‑8859‑1/‑15); system scripts sometimes still run with non‑UTF‑8 locales, exposing bugs.
    • Some software still mishandles multibyte UTF‑8 or non‑BMP code points.

BOM, utf-8-sig, and interoperability

  • Many Linux tools (including Python when using plain utf‑8) choke on UTF‑8 with BOM, even though it’s technically allowed and common from Windows tools.
  • Python’s utf-8-sig codec exists to handle optional BOMs, but is obscure in docs.
  • Debate:
    • One side wants BOMs “handled gracefully” by default on input.
    • Others resist: a BOM is also a real character (U+FEFF), and automatic stripping breaks exact round‑trip of text; also UTF‑8 BOMs are now generally discouraged.

String representation and Unicode subtleties

  • Clarifications:
    • Python str is a sequence of Unicode code points (not “UTF‑8 strings”); internal storage uses 1/2/4‑byte arrays depending on max code point, with cached UTF‑8 when needed.
    • This allows O(1) indexing but permits “ill‑formed” strings containing surrogate code points that cannot be encoded in standard UTFs.
  • Some consider this design flawed; others note that correct user‑visible “characters” are grapheme clusters anyway, so code‑point indexing is already a compromise.

Broader reflections

  • Several commenters view global locale defaults (for both numbers and text) as a long‑term design mistake.
  • Line endings (CRLF vs LF) and encodings are cited together as persistent cross‑platform friction that modern tools and languages are slowly, but not completely, ironing out.