The One Billion Row Challenge in Go: from 1m45s to 4s in nine solutions

Original Article ↗ Hacker News Discussion ↗

Optimizing Go code for the “1 Billion Row Challenge” — parsing a 13GB, 1‑billion‑line text file of temperature readings — shows how far careful profiling, custom data structures, and parallelism can push a straightforward program, from nearly two minutes down to about four seconds. Commenters compare these Go results with highly tuned Java, .NET, C++, Rust and even database- and GPU-based approaches, noting that JVM and .NET JIT/AOT toolchains and SIMD support can achieve even lower runtimes at the cost of extreme, non‑idiomatic code. Many emphasize that while these micro‑benchmarks are a fun way to explore performance limits, real-world solutions must balance raw speed with simplicity, robustness, portability, and the capabilities of higher-level libraries and databases.

Language performance comparisons

  • Thread centers on why the fastest Java and C#/.NET solutions beat the fastest Go ones, sometimes by factors of 2–4x.
  • Some argue Java and .NET benefit from very aggressive JIT/AOT, SIMD intrinsics, PGO, and decades of runtime tuning.
  • Others counter that Go can, in principle, use similar algorithmic tricks; the gap is more about implementation effort than inherent possibility.
  • There’s debate over claims that JVM can “often beat C”; skeptics argue this is only true in cherry‑picked cases.
  • Rust, C++, Swift, Dart, Node.js, and even R are mentioned via other leaderboards or examples, but comparisons are muddied by different hardware.

Optimization techniques and data structures

  • Key tricks: custom hash tables, memory-mapped IO, integer-only temperature parsing, loop unrolling, and careful use of unsafe memory access.
  • Discussion of potential further gains: stack-allocated arrays, reducing copy(), eager tries for station names, lookup tables for temperatures, and perfect hashing.
  • Several participants doubt large LUTs will beat simple arithmetic due to cache and memory-latency costs.

IO, caching, and benchmarking caveats

  • Many note that repeated runs keep the 13GB file in OS cache or RAM disk, so disk bandwidth isn’t the bottleneck.
  • Questions about Java solutions seemingly exceeding SSD throughput are answered by caching and in-RAM filesystems.
  • Some remark that parallelism gives large wins but isn’t directly comparable to single-threaded tools like cat.

Libraries, databases, GPUs, and high-level tools

  • People test or propose Polars, DuckDB, BigQuery, SQL databases, and even GPUs; these can get to tens of seconds or a few minutes with far higher-level code.
  • Some argue that for real systems, doing more work inside databases can be competitive and operationally simpler than hand-rolled code.

Real‑world relevance and Go-specific notes

  • Several emphasize the exercise is a “game”: real code would need robust parsing and error handling, trading speed for correctness.
  • Others focus on Go’s compiler limitations (weak SIMD story, modest PGO, less aggressive inlining) versus more mature Java/.NET toolchains, while noting Go’s strengths in simplicity and fast startup.
  • Profiling via Go’s pprof and techniques like disabling GC, locking OS threads, and using unsafe pointers are highlighted as useful but non-idiomatic.