Making C and Python Talk to Each Other

Original Article ↗ Hacker News Discussion ↗

Lua (and others) vs embedded Python

  • Several comments compare Python unfavorably to Lua as an embedded scripting language.
  • Lua is described as:
    • Much lighter and faster than Python, with trivial integration (drop-in source, minimal build friction).
    • Easy to sandbox: embedder decides what APIs exist, can limit file access, instruction count, and memory, making it attractive for games and untrusted addons.
    • Free of global state/GIL; multiple interpreters can coexist independently.
    • GC and C API are simpler (stack-based VM, few value types), hiding most memory-management complexity from the embedder.
  • Python embedding is seen as:
    • Heavier (full interpreter), historically hampered by global state and the GIL (especially pre-3.12), making multi-interpreter use problematic.
    • Much harder to sandbox and therefore risky for hostile code.
    • C API is considered fragile, especially around reference counting and garbage collection.

Why Python dominates (esp. AI/ML)

  • Many agree Python’s main strength is its ecosystem and packaging (pip, PyPI): “second best” at almost everything but with libraries for nearly anything.
  • For AI/ML, Python lets users call highly-optimized C/C++ (NumPy, PyTorch, etc.) without needing low-level expertise; productivity wins over raw speed.
  • Counterpoint: this division isn’t always clean. Large frameworks like PyTorch can place Python in the hot path (kernel launch overhead, distributed training, data loading), making peak performance harder.

Performance debates: C vs Python

  • C is “many magnitudes faster” for tight loops and low-level work, but several argue:
    • Syntax is mostly irrelevant; architecture (interpreted CPython, boxed integers, GIL) dominates Python’s slowness.
    • Non-experts often write C that’s slower than Python calling tuned libraries.
  • Others stress that language abstractions and runtime design (GC, exceptions, dynamic types) have real performance costs, contrasting C, C++, Python, and even Lisp examples.

Interop tooling and patterns

  • Alternatives to direct C API: pybind11, cffi, Cython, nanobind, Nim+nimpy, SWIG; some report large ergonomic gains migrating from SWIG.
  • Official Python docs on embedding are cited as a solid starting point.
  • One commenter describes a C raytracer wrapped with a small C API, then CPython bindings and higher-level Python wrappers, enabling a Blender plugin.

Visualization and C libraries in Python

  • Example: a C-based visualization engine packaged as a Python wheel via autogenerated ctypes bindings.
  • Claims orders-of-magnitude speedups vs Matplotlib for large interactive point clouds, while intentionally remaining a low-level rendering backend.

Critiques and API nits

  • The article is criticized for only covering C→Python calls despite its “making C and Python talk” title.
  • Low-level advice: prefer Py_BuildValue for building multiple return values, be careful with Py_XDECREF, and treat reference counting as a subtle, failure-prone area.