The Changing "Guarantees" Given by Python's Global Interpreter Lock

Original Article ↗ Hacker News Discussion ↗

Python’s Global Interpreter Lock (GIL) is being reconsidered, raising questions about what thread-safety guarantees Python actually offers and what has merely been an accident of CPython’s implementation. Commenters debate whether relying on the GIL for atomic operations is “broken” or de facto standard practice in an ecosystem without a formal language spec, and what obligations core developers have not to “break userspace” as they move toward optional GIL-less builds. Many see per-object locking and clearer documentation of what is truly atomic as a necessary but risky shift, trading some single‑threaded performance and implicit safety for better multicore utilization and more predictable concurrency semantics.

Scope of the GIL and Atomicity Guarantees

  • Many argue that treating “things the GIL makes look atomic” as truly thread‑safe is broken design; proper synchronization primitives should be used.
  • Others counter that when most of the ecosystem depends on a behavior, it becomes a de facto standard, making changes extremely hard (Hyrum’s Law, “don’t break userspace”).
  • Clarification: the GIL mainly protects interpreter/internal data, not user‑level race conditions. Code like x = self._next_id; self._next_id += 1 is not behavior‑wise thread‑safe today, just memory‑safe.
  • Some list/dict operations are explicitly documented as atomic (e.g., D[x] = y), and people rely on this; preserving such guarantees in a no‑GIL world is seen as crucial.

Plans and Concerns Around Removing the GIL

  • Future --disable-gil/nogil builds will use fine‑grained per‑object locks; typical list/dict mutations should not segfault and often remain atomic.
  • Example given: concurrent list.extend calls will yield either full A-then-B or B-then-A results, not interleaving, under nogil.
  • Overhead: extra locking is reported to slow single‑threaded code by roughly 5–10%; some worry that “lock on every mutable write” will be “slow”, others say uncontended locks are cheap.
  • There is concern about which operations will be guaranteed atomic and which won’t, especially where iterators and user code are involved. Better documentation of thread‑safety is expected but still “a lot of work”.

Python Semantics, Specification, and Implementation Details

  • Strong debate over whether Python has a “spec”:
    • One side: the Language Reference plus PEPs is a de facto specification distinguishing language guarantees from CPython‑only details (e.g., GIL, refcounting).
    • Other side: it’s descriptive, not a formal standard; the true reference is CPython itself, and GIL is part of Python’s semantics in practice.
  • Alternative implementations (PyPy, Jython, IronPython, MicroPython, etc.) deviate in various ways, reinforcing that some CPython behaviors (like the GIL) cannot be considered required language features.

Practical Impact and Use Cases

  • Some web and I/O‑bound developers report never being limited by the GIL; threading mainly hides I/O latency.
  • Others in CPU‑bound, data‑processing, and ML contexts find the GIL a major obstacle, resorting to multiprocessing with pickling overheads.
  • Suggestions include retaining a GIL‑compat mode and even bumping to “Python 4” for a GIL‑less model, reflecting the scale of semantic and ecosystem change.

Versioning Digression

  • Side discussion on Python’s 3.9 vs 3.13 style: version components are dot‑separated integers, not decimals.
  • This is framed as common, semver‑like structure, though Python’s “minor” releases can still introduce breaking changes.