You probably don't need to validate UTF-8 strings

Original Article ↗ Hacker News Discussion ↗

Whether programming languages should enforce that strings are valid UTF‑8 or treat text as arbitrary bytes is at the center of this debate. Commenters contrast Rust’s strictly validated UTF‑8 `str` with more permissive byte‑oriented approaches, weighing performance, simplicity of string APIs, and safety against problems like malformed filenames, broken normalization, and subtle regex behavior. Many conclude that strict validation is valuable at system boundaries and for high‑level text handling, but that low‑level tools and systems code often need flexible, encoding‑agnostic byte strings to avoid unnecessary overhead.

Scope of UTF-8 validation

  • One side argues you must validate if you do normalization, indexing/splitting by character, Unicode-aware regex, or interoperate with systems that assume valid UTF-8; otherwise length and semantics are ill-defined.
  • Others claim you can often defer validation or skip it: treat non‑UTF‑8 as raw bytes, use byte-oriented regex, and only validate/normalize at specific boundaries (e.g., before JSON output or as hash keys).
  • There’s a meta-point that scanning for invalid sequences at any point is already “validation,” just with different error handling (e.g., replacement vs failure).

Language string design (Rust, Go, Python, etc.)

  • Rust: str is guaranteed valid UTF‑8; this simplifies many operations and lets implementations assume correctness, but forces up-front validation or use of &[u8] for arbitrary data.
  • Some argue this is mostly “purity” plus minor performance wins; others say it’s crucial because it makes illegal states unrepresentable and centralizes robustness at the type boundary.
  • Go is cited as successful with “conventionally UTF‑8” strings that gracefully map invalid sequences to replacement characters.
  • Python strings are sequences of Unicode code points, not necessarily valid UTF‑8; surrogate issues show that “Unicode string” and “UTF‑8 encodable string” differ.
  • Several commenters favor byte strings as the fundamental type, with Unicode as an optional layer; others note ecosystems that strongly prefer UTF‑8 (web, many tools).

Semantics: equality, substrings, and length

  • Substring search in UTF‑8 is easier than in encodings like UTF‑16 due to self-delimiting code units, but normalization and combining marks mean byte-substring search often misses semantically equivalent text.
  • Canonical Unicode normalization is considered expensive and table-driven; there is “no cheap canonical UTF‑8.”
  • “String equality” is framed as context-dependent: byte equality, code-point equality (after normalization), locale-aware collation, visual equivalence, or application-specific notions (addresses, names).
  • Length is also context-dependent: bytes vs code points vs grapheme clusters vs rendered width.

File paths and non‑Unicode data

  • Debate on whether programs should insist on UTF‑8 file paths: some say it simplifies cross-platform handling; others argue many tools only need opaque byte sequences and rejecting valid but non‑UTF‑8 paths is user-hostile.
  • Strategies mentioned: using dedicated UTF‑8 path types, WTF‑8 for Windows surrogates, or keeping paths as raw bytes and only decoding when displaying.

Immutability and performance

  • Brief side thread: mutability gives O(1) in-place updates, while immutability can add log N overhead; counterpoint is that immutability can enable better global optimizations and parallelization, though benchmarks favor mutable designs today.