A 100x speedup with unsafe Python

Original Article ↗ Hacker News Discussion ↗

Meaning of “unsafe Python”

  • “Unsafe” here mostly means exposing and manipulating low-level memory layout (e.g., base pointers, strides), not full arbitrary pointer arithmetic like in C.
  • Some commenters initially conflate this with general type safety or static checks; others clarify the distinction between memory safety and type safety.

Python safety and CPython’s actual guarantees

  • Several posts note Python is generally type- and memory-safe at the language level (e.g., out-of-bounds and type errors raise exceptions).
  • Others emphasize CPython itself is not memory safe:
    • ctypes can dereference arbitrary addresses and segfault.
    • Mutating function __code__ objects in certain ways can crash the interpreter.
    • A linked repo shows memory-unsafe tricks using only built-ins.
  • There’s meta-discussion that “safety” is overloaded and context-dependent.

Performance tricks: numpy, strides, and ctypes

  • Core article trick: reinterpret image buffers via ctypes to avoid copies and get ~100x speedup for a specific pygame/OpenCV resize path.
  • Some suggest safer alternatives:
    • np.ascontiguousarray or advanced stride tricks; others report this doesn’t remove the main overhead in the benchmark.
    • Directly modifying strides and using numpy.lib.stride_tricks.as_strided is discussed; base-pointer shifts outside the original buffer are hard or impossible safely.
  • One commenter calls the title clickbait since the speedup is very specific (SDL + numpy interaction).

Contributing upstream vs local hacks

  • One camp says the right fix is a small OpenCV (or binding) patch so everyone benefits.
  • Counterpoint: handling this particular format correctly inside OpenCV is nontrivial, may not be accepted, and doesn’t help other libraries; “local” tricks can be valuable.

Other tools and “unsafe” options

  • ctypes, C extensions, and CFFI are widely recognized as “unsafe zones” in Python.
  • MicroPython’s “Viper” mode is cited as a constrained, C-like subset that can give 10–100x speedups while staying close to Python syntax.
  • Cython vs numba: Cython seen as less “magic,” good for explicit optimizations and wrapping C with numpy; numba better for JIT-ing existing Python.

Arrays, layout, and scientific ecosystems

  • Long discussion on row-major vs column-major arrays, strides, slicing, and views vs copies:
    • Numpy defaults to row-major but supports both; confusion over wording in the article.
    • Striding lets you avoid copies but adds complexity and potential performance pitfalls.
  • Broader debate on multidimensional array support in languages (Go, Rust, new languages) and how design choices impact performance and ergonomics.
  • Fortran and Matlab are argued to still be strong in certain HPC and engineering niches due to array-centric syntax, established libraries (BLAS/LAPACK), and tooling, even as Python/C/C++ gain ground.
  • Julia is mentioned as a modern array-heavy language achieving C/Fortran-like speed.

Language preferences and pedagogy

  • Some criticize overreliance on Python in education, comparing it to earlier “Java-only” cohorts.
  • Others defend Python for being easy to write and prototype in, then rewriting hot paths in C or another language when necessary.
  • Mojo, Nim, and others appear as “have-your-cake” contenders (Python-like syntax with performance), though Mojo’s closed compiler limits excitement for some.