RAG Is Simpler Than You Think

Original Article ↗ Hacker News Discussion ↗

Retrieval-augmented generation (RAG) is portrayed as less novel than its hype suggests, with many arguing it’s essentially traditional information retrieval plus an LLM and that teams often overcomplicate it with vector databases and embeddings. Commenters repeatedly emphasize that full-text search and BM25 remain powerful, cheaper, and easier to control for many real-world use cases, especially technical and code search, and that embeddings are often overestimated, tricky to tune, and operationally costly. There is also skepticism about AI-generated how‑to content around RAG, along with calls for clearer design patterns, better evaluations, and a “start simple, add complexity only when needed” approach.

What RAG Actually Is

  • Many comments say RAG is essentially “classic information retrieval + LLMs,” not something fundamentally new.
  • Several note that “RAG ≠ vector search”: any retrieval (grep, SQL, FTS, web search) whose results are fed into the LLM is RAG.
  • Others in the thread still implicitly treat RAG as “embeddings + vector DB,” showing ongoing confusion.

Embeddings vs Full‑Text / Keyword Search

  • Strong theme: full‑text search (FTS/BM25/Lucene/Elasticsearch/Postgres FTS) is underrated and should usually be tried first.
  • Arguments for FTS: simpler, deterministic, mature tooling, easier to debug, scales well, and often works better for technical or code-like queries.
  • Arguments for embeddings: good for underspecified or descriptive queries, synonyms, multi-language search, and when users don’t know exact terms.
  • Some report real-world disappointment with embeddings: semantic similarity weaker than expected, operational overhead high, reranking pipelines complex.
  • Others report good results from simple embedding-based semantic search or hybrid BM25+vector approaches.

Chunking, Scale, and Document Structure

  • Chunk size is repeatedly cited as critical; bad chunking can sink retrieval regardless of model.
  • Long documents (books, reports, PDFs) raise issues: token limits, references across chunks (“it” referring back), and exploding index size when embedding many overlapping segments.
  • There’s debate on whether moderately large chunks (512–1024 tokens) are usually enough to capture topic context; some think yes, others say relational info is still lost.

Use Cases: Code, Enterprise Docs, Multilingual

  • For code: several tools and teams report that simple grep or FTS plus smart agents often outperform RAG; some even removed retrieval entirely without users noticing.
  • For huge codebases and abstract “concept” lookups, retrieval can still help.
  • In enterprise/finance docs with heavy jargon and acronyms, BM25 can struggle; embeddings or cross-language setups can work better.
  • Embeddings are framed as a language-agnostic “common space” for cross-lingual search, but debugging is harder.

System Design, Agents, and Orchestration

  • Many see the real “dragons” not in embeddings but in retrieval design: freshness, access control, versioning, overlapping docs, evals, and ranking.
  • Agentic query rewriting on top of classic search is popular: let the LLM craft and iterate queries, then use stable lexical/FTS infrastructure underneath.
  • Some view RAG as an orchestration problem: mixing modules (exact search, vector, SQL, tagging, rules) depending on use case and quality requirements.

Costs, Tooling, and Practical Tips

  • One camp says “embed everything once, track changes, and store vectors in something like BigQuery,” assuming modest corpus sizes.
  • Others warn about vendor lock‑in and suggest picking portable tech, simple Postgres/BM25/pgvector, or SQLite FTS5 with LLM‑generated SQL.
  • There is frustration that many RAG how‑tos don’t tie methods to evals; people want benchmarks and guidance on when each retrieval recipe actually improves outcomes.

Skepticism, Hype, and Writing Style

  • Some commenters argue the article reads like LLM‑generated “AI slop”: overblown headings, shallow “what” lists with weak “why,” and hybrid recipes that feel machine‑stitched.
  • Others still find the content practically useful despite the style.
  • Broader backlash appears: fatigue with AI‑generated prose, claims that RAG is overused, “magic beans,” or “so 2024,” and calls to start with the simplest search that works and only add complexity when clearly needed.