gh-116167: Allow disabling the GIL

Scope of the Change / Current Status

  • Change comes from work to make the GIL optional (PEP 703).
  • Current PR requires compiling CPython with a special flag, then disabling the GIL via a runtime toggle.
  • This is explicitly an early, research‑stage step: intended to work reliably only for thread‑free programs.
  • Asyncio tests fail with GIL disabled; some simple threaded programs “seem” to work but are not guaranteed safe.

Single‑Thread vs Multi‑Thread Performance

  • Removing the GIL enables true multi‑core parallelism in multi‑threaded Python, useful for high‑thread workloads (e.g., ML, research).
  • For single‑threaded code, multiple commenters note a performance penalty from extra locking; PEP 703 cites a 5–15% slowdown.
  • Some hope more optimizations could reduce overhead for single‑threaded nogil builds over time.

Asyncio, Threads, and Compatibility

  • Several comments clarify: async I/O and OS threads are distinct; the current breakages are mostly in asyncio tests that mix threads and coroutines.
  • There is confusion in the thread about whether “any threaded code” breaks; consensus is: many existing threaded patterns, especially involving shared objects, are unsafe in this release.

Alternatives to Threads Today

  • multiprocessing is widely cited as a workaround for CPU‑bound tasks, but:
    • On Windows and macOS, lack of fork / use of spawn makes it slow and awkward.
    • Large memory footprints and complex shared data structures make process‑based parallelism painful.
  • Libraries like Ray help with multi‑process and shared memory but have their own limitations (e.g., immutable array‑focused objects).

Ecosystem & C Extensions Impact

  • Most native (C/C++) extensions implicitly rely on the GIL for thread safety (globals, unsynchronized list mutations, etc.).
  • Plan is: if an extension depends on the GIL, it will keep it enabled; in the long run, extensions must be updated to be nogil‑safe or explicitly GIL‑using.

Python vs Other Languages & Typing

  • Several commenters argue that if you want “Python + types + concurrency” today, Go, Rust, C#, Kotlin, Julia, Nim, or BEAM languages may be better fits.
  • Others counter that Python’s ecosystem (NumPy, PyTorch, ML/DS tooling) dominates, and nogil plus modern type checkers (e.g., mypy, pyright) make Python competitive for many use cases.
  • Debate continues over Python’s speed and typing model vs “modern” languages, with acknowledgment that Python will remain slower but can still improve significantly.