gh-116167: Allow disabling the GIL
Scope of the Change / Current Status
- Change comes from work to make the GIL optional (PEP 703).
- Current PR requires compiling CPython with a special flag, then disabling the GIL via a runtime toggle.
- This is explicitly an early, research‑stage step: intended to work reliably only for thread‑free programs.
- Asyncio tests fail with GIL disabled; some simple threaded programs “seem” to work but are not guaranteed safe.
Single‑Thread vs Multi‑Thread Performance
- Removing the GIL enables true multi‑core parallelism in multi‑threaded Python, useful for high‑thread workloads (e.g., ML, research).
- For single‑threaded code, multiple commenters note a performance penalty from extra locking; PEP 703 cites a 5–15% slowdown.
- Some hope more optimizations could reduce overhead for single‑threaded nogil builds over time.
Asyncio, Threads, and Compatibility
- Several comments clarify: async I/O and OS threads are distinct; the current breakages are mostly in asyncio tests that mix threads and coroutines.
- There is confusion in the thread about whether “any threaded code” breaks; consensus is: many existing threaded patterns, especially involving shared objects, are unsafe in this release.
Alternatives to Threads Today
multiprocessingis widely cited as a workaround for CPU‑bound tasks, but:- On Windows and macOS, lack of
fork/ use ofspawnmakes it slow and awkward. - Large memory footprints and complex shared data structures make process‑based parallelism painful.
- On Windows and macOS, lack of
- Libraries like Ray help with multi‑process and shared memory but have their own limitations (e.g., immutable array‑focused objects).
Ecosystem & C Extensions Impact
- Most native (C/C++) extensions implicitly rely on the GIL for thread safety (globals, unsynchronized list mutations, etc.).
- Plan is: if an extension depends on the GIL, it will keep it enabled; in the long run, extensions must be updated to be nogil‑safe or explicitly GIL‑using.
Python vs Other Languages & Typing
- Several commenters argue that if you want “Python + types + concurrency” today, Go, Rust, C#, Kotlin, Julia, Nim, or BEAM languages may be better fits.
- Others counter that Python’s ecosystem (NumPy, PyTorch, ML/DS tooling) dominates, and nogil plus modern type checkers (e.g., mypy, pyright) make Python competitive for many use cases.
- Debate continues over Python’s speed and typing model vs “modern” languages, with acknowledgment that Python will remain slower but can still improve significantly.