Show HN: I've built a locally running Perplexity clone

Original Article ↗ Hacker News Discussion ↗

Project architecture and capabilities

  • Tool combines a local LLM with:
    • A meta-search engine (searxng) for web results.
    • A vector database for retrieval-augmented generation (RAG).
  • Workflow: LLM issues search queries → top N pages scraped and chunked → chunks embedded into a vector DB → LLM queries DB to answer.
  • Positioned as a Perplexity-like UX, but fully local LLM; actual web index still comes from external engines via searxng.
  • Demo runs on a 12GB VRAM GPU; LLM backend via Ollama / llama.cpp. Some see it as easily adaptable to corporate/internal search (e.g., Confluence).

Models, quantization, and hardware

  • Several 7B models are recommended (e.g., Hermes and Starling variants) for structured text and chat.
  • Quantized GGUF models discussed:
    • Higher quantization (Q5, Q6_K) generally preferred if RAM allows.
    • Trade-off between quant level and context length under RAM limits.
  • Users report good performance on M1/M2 Macs (8–32GB, often Q4–Q5 7B models); CPU-only use is possible but much slower.

Feature ideas and extensions

  • Requests to:
    • Restrict search to whitelisted/blacklisted domains or categories.
    • Work offline against locally mirrored sites, PDFs, books, and research papers.
    • Integrate with internal corporate data and other tools (e.g., agents, Plandex, Crew-like multi-agent setups).
  • searxng already supports site whitelists/blacklists; exposing this in the UI is suggested.

Implementation details and prompts

  • LLM decides when to use tools based on prompt format; function-like “Action: ...” patterns trigger web search or vector DB calls.
  • Main latency sources: generating embeddings for scraped pages and reading vector matches.
  • HTML is sanitized but not deeply cleaned; some noisy content likely persists.
  • Prompts are iteratively refined to fix behavioral issues, highlighting how much performance depends on prompt engineering.

Perplexity, value, and ecosystem discussion

  • Some claim Perplexity initially relied heavily on off-the-shelf models and simple “summarize top N results”-style logic; others credit it for good integrations (e.g., YouTube, Reddit).
  • Debate over why such products attract >$100M funding versus what a motivated individual can build.
  • Broader argument about future of AI:
    • One side predicts more local inference for privacy, latency, and offline use.
    • Another argues centralized, large models on specialized hardware will remain dominant; both likely to coexist.