So you wanna build a local RAG?

Original Article ↗ Hacker News Discussion ↗

Developers exploring “local RAG” (retrieval-augmented generation run entirely on personal or on-prem hardware) are weighing trade-offs between semantic vector search and simpler full‑text or keyword approaches. Many report that hybrid strategies—combining embeddings, BM25, and careful chunking or contextual summarization—often work best, but only when validated with realistic evaluation data that matches how end users actually query information. The thread also highlights practical constraints such as hardware limits, vector database maintenance, document parsing complexity (especially PDFs and tables), and the surprising effectiveness of plain search plus an LLM “agent” for many use cases.

Semantic chunking and document structure

  • Several comments stress that embedding whole documents hurts performance; semantic chunking plus added context about each chunk’s role in the doc can dramatically improve retrieval.
  • Anthropic-style “contextual retrieval” (generate summaries/metadata around chunks) is cited as particularly effective.
  • Some wonder if GraphRAG / knowledge-graph approaches could better capture cross-document structure, similar to personal knowledge tools.

Lexical vs semantic search (vectors) debate

  • One camp argues you can skip vector DBs: full-text search (BM25, SQLite FTS, grep/rg, TypeSense, Elasticsearch) plus an LLM-driven query loop often works “well enough,” is cheaper, simpler, and avoids chunking issues.
  • Others report that pure lexical search degrades recall, especially when users don’t know exact terminology; multiple search iterations inflate latency.
  • A common framing: lexical search gives high precision / lower recall; semantic search gives higher recall / lower precision.
  • Many advocate hybrid systems (BM25 + embeddings, fusion, reranking) as the current best practice, though added engineering complexity is questioned.

Evaluation and real-world usage

  • Strong emphasis on proper evals: build test sets of [query, correct answer], use synthetic Q&A, and compare BM25 vs vector vs hybrid configurations.
  • People note dev-created queries are biased (they “know the docs”); real users phrase things differently, revealing much poorer performance.
  • Some propose automated pipelines and LLM judges to continuously score RAG changes.

Local models and infra

  • One view: running a local LLM is overkill; keeping only docs and vector DB local is already a big win.
  • Others say consumer hardware (16–32GB GPUs or high-RAM laptops) can run substantial local models and that medium-sized orgs can self-host if they value privacy.

Practical challenges and tooling

  • Document parsing (especially PDFs with tables, images, multi-page layouts, footnotes) is described as a major unsolved pain point, often more limiting than retrieval method.
  • Various tools/stacks are mentioned: llama.cpp-based apps, Elasticsearch, Chroma, sqlite-vec, local RAG GUIs, Nextcloud + MCP, and open-source RAG frameworks with benchmarks.
  • Some highlight language issues: many embedding models are English-only; multilingual or language-specific models and leaderboards are needed for non-English RAG.