Benchmarking 20 programming languages on N-queens and matrix multiplication

Benchmark goals and methodology

• Benchmarks cover N-queens, matrix multiplication, sudoku, and bed coverage with mostly hand-written, “naive” implementations, often inspired by Rosetta-style examples.
• Some see this as useful for gauging baseline language/VM performance when you must implement a new algorithm yourself.
• Others argue the code is suboptimal and non-idiomatic in several languages, so results may say more about particular implementations than the languages.
• There is debate over whether to include JIT warmup and startup time; some argue “from CLI to result” is realistic, others want warmed-up JITs or to also count compile time for AOT languages.

Libraries vs “pure language” performance

• A major thread disputes excluding libraries like NumPy, BLAS, or specialized tensor libs.
• One side: benchmarks should compare languages without external C/Fortran libraries; otherwise you just measure FFI overhead.
• Other side: idiomatic numerical Python, C#, etc. always use such libraries; pure-language matmul is unrealistic and misrepresents real-world performance.
• It’s noted that essentially all high-performance matmul in any language ultimately relies on heavily optimized libraries, often with assembly, and that Python is typically “fast” only when calling out to them.

Matrix multiplication difficulty and BLAS

• Several comments stress that truly BLAS-level matmul requires careful tiling, SIMD, cache/register-aware kernels, and sometimes assembly; compilers won’t reach >90% of peak with a simple triple loop.
• Some claim you can match or beat BLAS for specific shapes/sparsity patterns with C++/Nim-level control; others push back, reporting experiments where hand-written code lagged far behind OpenBLAS except with significant effort.
• A Nim project is cited with benchmarks showing performance comparable to OpenBLAS and MKL on multiple CPUs, using generated SIMD microkernels and custom threading.

Language-specific observations

• Rust: initial matmul was slow due to Vec<Vec> layout and bounds checks; iterator-based or statically sized implementations now match C.
• C#: early matmul used rectangular arrays (extra multiplications per access). Switching to array-of-arrays brings performance closer to Java; further gains expected from SIMD (Vector<T>/System.Numerics.Tensors).
• Swift: matmul can match C/Rust after optimization, but sudoku remains slow due to many heap allocations and lack of static arrays.
• Julia: initial matmul used wrong memory orientation and disabled SIMD; fixed code performs near C.
• Mojo: about 2× slower than C on matmul, seen by some as acceptable given ergonomics, though others find it verbose compared to Julia.

JIT, startup, and hardware

• Some argue JIT languages are disadvantaged by cold runs; others counter that many real-world uses (CLIs, builds) are short-lived, so cold performance matters.
• ARM big.LITTLE systems raise questions about whether benchmarks run on performance vs efficiency cores.
• Mono-, multi-threaded, and vectorization details (e.g., OpenBLAS threading, AVX/AVX-512 usage, bounds checks) significantly affect observed differences.

Charts and alternative metrics

• Stacked bar charts obscure comparisons when a few languages (PHP, Ruby, Perl, pure CPython) are extremely slow; suggestions include separate charts, log scales, or ops/sec.
• Some want additional metrics like gzipped source size or LOC to reflect expressiveness and code complexity; others note the difficulty of defining a fair, human-centric size metric.
• Multiple commenters point to larger benchmark suites (e.g., the Computer Language Benchmarks Game) and request more real-world tasks (file I/O, JSON parsing, servers).