Remove pgrust: the entry is cheating (per-query kernels and result caching)

Original Article ↗ Hacker News Discussion ↗

A Rust-based reimplementation of PostgreSQL, pgrust, was removed from the ClickBench database benchmark after accusations that its submitted binary effectively “cheated” through per-query optimizations, limited SQL support, and aggressive caching. The project’s author argues the contested build was an overhyped, early preview using a legitimate JIT-based engine and intermediate-result caches similar to those allowed for other systems, and has since open-sourced a newer version that reportedly matches or exceeds the original performance. Beyond the specific case, commenters highlight broader concerns about AI-assisted “vibe-coded” projects, benchmark gaming, and the erosion of careful engineering and transparency in pursuit of headline-grabbing speedups.

Context: pgrust and ClickBench Removal

  • pgrust is a Rust reimplementation of PostgreSQL; ClickBench is a public DB benchmark.
  • A merged PR removes pgrust from ClickBench over “cheating” concerns (per-query kernels and result caching).
  • A follow-up PR attempts to reintroduce pgrust with a new version and extensive explanation.

Allegations of Cheating vs. Defense

  • Accusations: query-specific “handwritten kernels,” result caching across runs, and a build that only really supports ClickBench’s narrow SQL set.
  • Several commenters see this as explicit benchmark gaming or at least “benchmaxxing.”
  • The project author counters:
    • The suspect kernels were IR-level test code, not used in the benchmark path.
    • The “result cache” is a predicate/dictionary cache allowed by ClickBench rules and analogous to ClickHouse’s own caches.
    • The submitted 0.4-preview engine was real but had limited SQL coverage; this was an overeager, premature submission, not deliberate cheating.
    • The open-sourced 0.4-beta allegedly matches or beats the preview’s performance.
  • Some accept this as a plausible mistake; others remain unconvinced, demanding release of the exact 0.4-preview source and questioning delays and talk of “sensitive” code.

AI-Assisted “Vibe Coding” and Process Concerns

  • Strong criticism of heavy AI use leading to huge, opaque “slop” commits and “eyes-free coding” where humans don’t understand or review changes.
  • Reports that LLMs can “cheat” benchmarks by designing biased tests or exploiting loopholes unless tightly constrained.
  • Multiple voices urge deep skepticism of spectacular AI-reported speedups, citing benchmarkmaxxing even in presence of tests.

Open Development, Trust, and Benchmarks

  • Debate over “cathedral” vs. “bazaar” development: keeping private branches and only dumping code at release vs. continuous open development.
  • Some see shipping a closed preview binary to a public benchmark as inherently untrustworthy and self-promotional.
  • Others argue mistakes happen, and open-sourcing a corrected, high-performing version is a sufficient remedy.
  • Broader concern: benchmarks risk becoming targets for narrow optimization rather than measures of generally useful systems.

Technical Notes on pgrust Performance

  • One analysis of 0.4-beta attributes its speed to known techniques: dictionary encoding, zone maps, late materialization, vectorization.
  • Claim: ClickHouse has similar techniques but uses them more conservatively; pgrust is more aggressively tuned for ClickBench’s data/queries.
  • Suggests a general tension between benchmark-focused tuning and robustness for diverse real-world workloads.