Show HN: Turn any website into a knowledge base for LLMs

Original Article ↗ Hacker News Discussion ↗

A new service promises to turn any public website into a searchable knowledge base for large language models by crawling pages, chunking and embedding their content, and exposing it via an API for RAG and semantic search. Commenters probe its technical choices (serverless stack, Pinecone, heading-based chunking), pricing, and limitations such as handling logins, PDFs, or images, while sharing open source alternatives and DIY setups. A recurring theme is the ethical and legal tension around web scraping—robots.txt compliance, user-agent transparency, GDPR, and data ownership—alongside skepticism about the long‑term viability of third‑party crawlers as more sites lock down content or build their own AI interfaces.

What the tool does

  • Crawls provided URLs/domains, extracts content, embeds it into a vector DB, and exposes an API for RAG and semantic search.
  • Works with most sites, including GitBook-style documentation.
  • Supports grouping multiple websites into a “collection” that can be queried as one KB.
  • Attempts to auto-discover sitemaps; explicit sitemap submission is on the roadmap.
  • Currently cloud-only; no on-prem/self-hosting option.

Tech stack and implementation details

  • Built with serverless Laravel plus Cloudflare and AWS functions; Pinecone for vector storage.
  • Pages are chunked based on heading hierarchy (h1, h2, etc.), not <section> tags; each chunk keeps its heading context.
  • Respects robots.txt according to the author, with plans to document user agents and behavior more clearly.
  • No image embeddings yet; considered a future feature.

Use cases, questions, and feature requests

  • Interest in applying it to PDFs, forums, dev docs, and internal sites; some ask about sitemaps and WARC ingest.
  • Logins / gated content: not supported unless you own the site; details for authenticated crawling are unclear.
  • Users want:
    • Longer, richer answers and more conversational behavior.
    • Prompt customization.
    • Export of data (vectors vs. text chunks) and possibly no-code formats like PDF.

RAG quality, models, and limitations

  • Underlying LLM and specific RAG configuration are not clearly described; several ask about models and hallucination rates.
  • One user notes the demo chat feels limited and more like a proof-of-concept than the main product (the API).

Pricing, business viability, and reliability

  • Interest in reasonably priced non-enterprise tiers; some explicitly say they would pay for a solid service.
  • Enterprise pricing is “contact us,” which some find ominous or vague.
  • Others argue the niche is easy to build yourself and may be short-lived; counter-voices defend it as useful and non-trivial to productize.
  • Reports of 404s, internal server errors, and broken confirmation emails during launch.

Ethics, scraping, and broader trends

  • Debate over ethics:
    • Concerns about GDPR, consent, and inability to easily block or audit this specific crawler.
    • Criticism of disguising the crawler as a regular browser.
    • Comparisons to wider AI scraping issues and to Clearview-like data use.
  • Some see services like this as accelerating bot-blocking and gating of websites; others suggest microtransactions or built-in site AI as future directions.
  • Discussion branches into IP, attribution, and whether small content owners gain or lose from AI-driven reuse of their data.

Alternatives and DIY approaches

  • Multiple users mention building similar systems themselves with Playwright, OCR, various embeddings, and open-source RAG stacks.
  • Several open-source tools and libraries are referenced (e.g., Vectara-based apps, SQLite-based RAG, Ollama + LangChain, web UIs), indicating a rich DIY ecosystem around “chat with any site” functionality.