Training a 4B model to produce 81% faster query plans than Postgres

Original Article ↗ Hacker News Discussion ↗

A blog post claims a 4‑billion‑parameter language model can generate PostgreSQL query plans that run about 81% faster on a small, in‑memory benchmark, prompting debate over whether LLMs belong in core database optimization. Commenters question the practicality of retraining models, the omission of planning time from performance claims, and the risk of overfitting to narrow workloads, while noting that traditional cost‑based and machine‑learning heuristics already struggle with changing data distributions. Many see potential for offline or hybrid use—e.g., generating hints for problematic queries—rather than replacing mature, deterministic planners in high‑throughput, real‑world systems.

Correctness and what the LLM actually controls

  • Several ask how you know the LLM-produced plan still computes the right query.
  • Clarification: the LLM doesn’t rewrite SQL; it only nudges planner settings/hints. Postgres still verifies plan validity and falls back if hints don’t correspond to a legal plan.
  • Some suggest equivalence checking or proof tools, but others note that fully proving equivalence is undecidable/NP-hard in general.

Determinism, statistics, and workload drift

  • Query plans are not deterministic; they depend on statistics, parameter values, and data shape.
  • Real-world workloads often see sudden plan regressions when stats change or rare cardinality patterns show up.
  • Hints or learned plans can become invalid when data distributions or workloads drift, limiting the value of “once-and-done” offline optimization.

Performance claims and benchmark realism

  • Multiple commenters question the 81% speedup:
    • Dataset is small (8 GB), fits in memory, queries are warmed, read-only SELECTs.
    • Concern about overfitting to this environment and lack of evidence on large, evolving OLTP/HTAP systems.
    • Some note that mis-tuned Postgres parameters (e.g., random_page_cost, missing indexes/statistics) alone can explain big gaps.
  • Others point out that the project ignores planning latency; a practical planner must improve “planning + execution” under live constraints.

LLMs vs other approaches

  • Many argue classic ML or specialized neural nets (AlphaGo-style heuristics, GNNs) would be more appropriate than a general LLM.
  • Existing tools like cost-based optimization, Postgres histograms, GEQO, and learned heuristics in compilers are cited as more targeted precedents.
  • Some see LLMs as overkill and hard to debug; others view them as promising for offline exploration and distillation into smaller models.

Operational risk and usage models

  • Skepticism about adding a 4B-parameter model into production-critical DB paths, especially at high QPS.
  • More enthusiasm for offline or testing-time use: clone prod, analyze slow queries, generate hints, commit them to version control, and verify via tests.
  • Concerns remain about non-determinism, regression risk, and skill mismatch for DB teams managing LLM-based components.

Other themes

  • Discussion of potential GPU acceleration for joins/sorts versus smarter planners.
  • Mentions of adaptive query plans (mid-flight plan switching) as the longer-term “endgame.”
  • Side debate on distillation ethics and the broader AI ecosystem, plus some praise for the write-up’s clarity and visual quality.