Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

Original Article ↗ Hacker News Discussion ↗

Model size, storage, and browser limits

  • Models (e.g., Llama 3) are 4–5GB and downloaded on first use, then cached in IndexedDB; browser may later evict them.
  • Some users were surprised browsers can store multi‑GB blobs, having assumed ~400MB limits.
  • There’s interest in resources explaining modern browser quota behavior.
  • Concerns raised that if this pattern spreads, users may repeatedly download large models for many sites, filling disks.

WebGPU support and compatibility

  • App runs only with WebGPU (practically Chrome/Edge desktop; limited or experimental on Firefox, Linux, iOS Safari).
  • Several users report “Cannot find WebGPU” or “Cannot find adapter” errors; workarounds involve enabling flags or OS GPU settings, with mixed success.
  • Some devices (e.g., Pixel 6/6a, M1 Macs) run it surprisingly well; others hit driver/platform limitations.

Performance, UX, and bugs

  • Many report fast load and good token speed; some compare favorably to cloud GPT‑4 for everyday prompts on consumer GPUs.
  • Others argue browser-based LLMs still struggle with stability, lag, and hardware variability for “serious” UX.
  • A bug where resetting the chat mid-generation feeds partial output back into the model produces surreal “self‑hallucination”; users find it amusing and suggest a “mode” around it.
  • Requests for features: chat history, wider text layout, local document/knowledge integration (e.g., via retrieval systems).

Model quality and behavior

  • Llama 3 8B is seen as strong for its size but not flawless: some users experience poor contextualization (e.g., song-naming prompt derailing into repeated famous titles and incoherent backreferences).
  • Smaller models like Phi‑1.5 show odd “inner dialogue” and appear less usable; Phi‑3 and other 1–2B models are recommended as better small options.
  • Confusion over “web browsing”: models can convincingly hallucinate page summaries without actual network requests.

Local-first, privacy, and ecosystem ideas

  • Strong enthusiasm for private, local-first AI that avoids sending sensitive data to cloud providers.
  • Suggestions include:
    • browser- or OS-level model managers shared across sites,
    • extensions/APIs exposing a local LLM service to web apps,
    • deeper integration into browsers (e.g., standardized LLM APIs, permission prompts like camera/mic).