gh-116167: Allow disabling the GIL
Work on PEP 703 to make Python’s Global Interpreter Lock (GIL) optional is advancing, with an experimental CPython build that can be compiled to run without the GIL for thread‑free programs. Commenters highlight the potential for true multi-core parallelism and simpler high-performance Python—especially for workloads that are currently forced into multiprocessing or C/C++—but note current breakages (notably in asyncio and threaded code) and a 5–15% slowdown for single‑threaded performance. The change is framed as a long-term, incremental shift that will require updates to native extensions and tooling, while reigniting comparisons with alternatives like Go, Rust, and Julia for typed, concurrent back-end development.
Scope of the Change / Current Status
- Change comes from work to make the GIL optional (PEP 703).
- Current PR requires compiling CPython with a special flag, then disabling the GIL via a runtime toggle.
- This is explicitly an early, research‑stage step: intended to work reliably only for thread‑free programs.
- Asyncio tests fail with GIL disabled; some simple threaded programs “seem” to work but are not guaranteed safe.
Single‑Thread vs Multi‑Thread Performance
- Removing the GIL enables true multi‑core parallelism in multi‑threaded Python, useful for high‑thread workloads (e.g., ML, research).
- For single‑threaded code, multiple commenters note a performance penalty from extra locking; PEP 703 cites a 5–15% slowdown.
- Some hope more optimizations could reduce overhead for single‑threaded nogil builds over time.
Asyncio, Threads, and Compatibility
- Several comments clarify: async I/O and OS threads are distinct; the current breakages are mostly in asyncio tests that mix threads and coroutines.
- There is confusion in the thread about whether “any threaded code” breaks; consensus is: many existing threaded patterns, especially involving shared objects, are unsafe in this release.
Alternatives to Threads Today
multiprocessingis widely cited as a workaround for CPU‑bound tasks, but:- On Windows and macOS, lack of
fork/ use ofspawnmakes it slow and awkward. - Large memory footprints and complex shared data structures make process‑based parallelism painful.
- On Windows and macOS, lack of
- Libraries like Ray help with multi‑process and shared memory but have their own limitations (e.g., immutable array‑focused objects).
Ecosystem & C Extensions Impact
- Most native (C/C++) extensions implicitly rely on the GIL for thread safety (globals, unsynchronized list mutations, etc.).
- Plan is: if an extension depends on the GIL, it will keep it enabled; in the long run, extensions must be updated to be nogil‑safe or explicitly GIL‑using.
Python vs Other Languages & Typing
- Several commenters argue that if you want “Python + types + concurrency” today, Go, Rust, C#, Kotlin, Julia, Nim, or BEAM languages may be better fits.
- Others counter that Python’s ecosystem (NumPy, PyTorch, ML/DS tooling) dominates, and nogil plus modern type checkers (e.g., mypy, pyright) make Python competitive for many use cases.
- Debate continues over Python’s speed and typing model vs “modern” languages, with acknowledgment that Python will remain slower but can still improve significantly.