Ask HN: How do I train a custom LLM/ChatGPT on my own documents in Dec 2023?

Hacker News Discussion ↗

Developers trying to “train” ChatGPT-style models on their own documents increasingly find that full fine-tuning is rarely needed or cost-effective; instead, most practical solutions rely on retrieval‑augmented generation (RAG), where external data is indexed and injected into prompts at query time. Participants compare frameworks such as LlamaIndex, LangChain, Haystack, and various hosted services (AWS Bedrock, Azure AI Studio, OpenAI Assistants, Notion, PrivateGPT, etc.), weighing ease of use, cost, and openness against issues like context-window limits, chunking strategy, hallucinations, and vector database choices. A recurring theme is that RAG works well for many small‑to‑medium corpora and enterprise search/chat use cases, while very large or highly proprietary datasets may eventually need more advanced approaches like continual pretraining or hybrid RAG + fine‑tuning.

What “training on your documents” usually means

  • Many commenters stress that most products claiming to “train on your docs” are actually doing RAG (retrieval‑augmented generation), not true fine‑tuning or pre‑training.
  • RAG flow: ingest documents → chunk + embed → store in vector DB → on each query, retrieve relevant chunks → stuff into LLM prompt.
  • True fine‑tuning on raw documents alone is said to mostly teach style/patterns, not reliable factual recall; Q&A style datasets are recommended if fine‑tuning is used.
  • Some argue RAG is the correct general approach; a minority call RAG “fundamentally flawed” or suitable mainly for smaller datasets.

Cloud and managed solutions

  • AWS Bedrock: knowledge bases (RAG) plus “continuous pre‑training”; seen as powerful but expensive, especially custom models and OpenSearch.
  • Alternatives suggested: use pgvector/RDS instead of OpenSearch; Pinecone as cheaper vector store.
  • Other hosted options mentioned: Amazon Q, Azure AI Studio + Semantic Kernel, OpenAI Assistants API (built‑in RAG), Notion Q&A, NotebookLM, Office Copilot‑style builders, various SaaS tools.

Open‑source and local stacks

  • Popular RAG frameworks: LlamaIndex (widely praised), LangChain (criticized as bloated/unstable), Haystack, Langroid, txtai, embedchain, Buster.
  • Turnkey/local apps: PrivateGPT, h2oGPT, Gpt4All, Verba, anything‑llm, Khoj, Cheshire Cat, GPT Researcher, secure RAG app layers.
  • Many run local models via Ollama, llama.cpp, or similar, often using models like Mistral or Mixtral.

Technical challenges discussed

  • Chunking strategy and metadata are seen as critical to RAG quality; naive fixed‑size chunks often underperform.
  • Scaling issues: with large corpora, only a few chunks enter the context window; long‑context models help but have “lost in the middle” concerns.
  • Hybrid retrieval (vector + BM25/lexical) is recommended, especially for acronyms and domain jargon.
  • Hallucinations remain unsolved; best mitigations are tight prompts, retrieval constraints, and evaluation/monitoring.
  • Multi‑language RAG relies on multilingual embedding models; typos/OOV terms are partially handled via subword tokenization and embedding similarity.

Costs, hardware, and practicality

  • Hosted fine‑tuning and custom models can cost thousands per month, seen as prohibitive for hobbyists.
  • Off‑the‑shelf models + RAG + cheaper vector stores are presented as the pragmatic path for most use cases.