The One Billion Row Challenge

Original Article ↗ Hacker News Discussion ↗

A Java-centric “One Billion Row Challenge” to compute min/mean/max temperatures from a 12 GB, 1‑billion‑line text file has become a showcase for extreme performance tuning and systems knowledge. Participants debate whether the task is I/O- or CPU-bound, trading techniques like memory mapping, custom number parsing, SIMD, multi-threading, and exploiting OS page cache behavior, while also arguing over fair benchmarking methods and rules that forbid dataset-specific hacks. Many contrast hand‑optimized Java with C, Rust, SQL databases, awk, and pandas, using the exercise to explore how modern hardware, runtimes, and data-processing tools behave under heavy, but conceptually simple, workloads.

Scope of the challenge & languages

  • Although framed as a Java challenge, many participants discuss and share implementations in C/C++, Rust, Go, .NET, Python, Perl, Awk, R, SQL, ClickHouse, DuckDB, Pinot, and others.
  • Some wish other JVM languages were allowed officially, while others note that “no external dependencies” rules out things like Hadoop or Pandas.

IO, caching, and file access strategies

  • A central theme is whether the task is IO‑bound or CPU‑bound.
  • Because the 12 GB file is run 5 times on a 32 GB machine and caches aren’t flushed, later runs effectively operate from page cache, making parsing and aggregation the main bottleneck.
  • Approaches debated: reading into RAM vs mmap, buffered IO vs O_DIRECT, io_uring, small fixed buffers, and handling lines that cross chunk boundaries.
  • Some argue relying on OS cache policy makes the benchmark less “pure”, others say it reflects real deployments.

Correctness constraints & overfitting concerns

  • Station names must be arbitrary UTF‑8 (no ;), up to 100 chars; temperatures are −99.9 to 99.9 with exactly one decimal.
  • This enables integer math (scale by 10) and simpler parsing than full floating‑point.
  • Several fast solutions were found to misuse hash functions or rely on the specific dataset and were removed; there is debate over what counts as unfair “overfitting”.
  • Ideas appear for dataset randomization or hidden seeds to prevent tuning to one file.

Benchmark methodology debate

  • The organizer discards fastest and slowest of 5 runs and averages the middle 3.
  • One side argues the minimum time best reflects algorithmic potential without noise.
  • Others counter that trimmed means better approximate real‑world performance under system noise, VM contention, and GC effects.

Algorithmic and implementation strategies

  • Common suggestions: stream processing with per‑station aggregates, integer temperatures, hand‑rolled number parsing, SIMD for delimiter search and parsing, minimizing allocations, and careful hash map design.
  • Some explore exotic ideas like custom state machines, perfect hashes, or specialized SIMD stateful parsers, though feasibility is debated.

Use of databases and analytics engines

  • Several participants demonstrate SQL engines (PostgreSQL, ClickHouse, DuckDB, Pinot, kdb+/q) solving the query very concisely but often more slowly, especially including ingest.
  • There is discussion about storage bloat, index strategies, array tricks, parallel query plans, and COPY vs INSERT for bulk loading.

Java, build tools, and language comparisons

  • Opinions on Java are mixed: some praise its performance and tooling; others complain about verbosity and boilerplate.
  • Long sub‑threads compare Maven vs Gradle vs Cargo/Go tools, incremental compilation behavior, and perceived build slowness.
  • Some note that while low‑level languages can be fastest, naive Java often beats naive C/C++ due to safer defaults and JIT optimizations.