UTF-8000: Unlimited UTF-8
A proposal called UTF-8000 extends UTF-8 to encode arbitrarily large integers while remaining backward compatible with existing UTF-8 text, raising the prospect of effectively “unlimited” code points beyond today’s 21‑bit Unicode range. Commenters debate whether this extra space is useful—suggesting applications from richer emoji and private-use glyphs to LLM token schemes—versus simply unnecessary complexity given that most of Unicode’s current space is still unused and tooling is deeply tied to UTF‑16/UTF‑8 as standardized. The thread also revisits long‑standing issues around encodings, such as self-synchronization, BOMs, legacy code pages, CJK efficiency, and how much new formats should care about constraints from unsafe or legacy environments.
Overall view of UTF-8000
- Seen as a clever, fun extension of UTF-8 that allows arbitrarily large code points while remaining byte-compatible with existing UTF-8 for current Unicode.
- Several people describe it as an elegant hack or natural extension of the original (pre-2003) UTF-8, which once allowed 5–6-byte sequences.
- Others argue it’s unnecessary given the vast unused Unicode space and the practical limit imposed historically for UTF‑16 compatibility.
Usefulness vs. Complexity
- Skeptics compare it to over-engineering (“IPv6 → IPv50”, “TIME CUBE for encoding nerds”) and argue we’ll never need that many characters; Unicode is a finite catalog, not a generic data container.
- A quoted letter from a UTF‑8 co‑author calls the 7–8 byte extensions “not useful” and notes that the 8-byte form forces scanning the whole sequence to know its length.
- Concerns are raised about arbitrary-length code points complicating buffer management and creating denial-of-service risks, though others counter that modern memory-safe approaches can handle this.
Properties and trade-offs
- Discussion of self-synchronization: UTF‑8000 is argued to preserve the “resync from any byte” property, but it likely breaks some useful homomorphisms (e.g., simple substring search on byte sequences).
- Some emphasize that even current Unicode already allows arbitrarily long grapheme clusters, so fixed-width “characters” are illusory anyway.
- Others praise UTF‑8’s design: ASCII-compatibility, self-synchronization, and the ability to do many operations on raw bytes.
Relation to other encodings / uses
- Historical context: UTF‑8’s range was cut from 31 to 21 bits to match UTF‑16, which some see as a mistake that ossified a too-small limit.
- Long side discussion on BOMs, UTF‑16, legacy encodings, and a wish that the world had standardized on UTF‑8 earlier.
- Some suggest that if we truly outgrow Unicode, a new serialization model or a non-self-synchronizing “re-embedding” of Unicode might be better than endlessly extending UTF‑8.
“What could we do with the space?”
- Playful ideas: fully customizable emojis, encoding images or scripts in single code points, “Unicode-as-a-character,” and even embeddings or LLM tokens.
- Others note that private-use ranges and grapheme clusters already allow many custom schemes, so UTF‑8000’s extra range may be more theoretical than practical.