Show HN: I built a free in-browser Llama 3 chatbot powered by WebGPU

Original Article ↗ Hacker News Discussion ↗

Running large language models entirely in the browser via WebGPU is starting to feel practical, as users report surprisingly fast performance with quantized versions of Llama 3 and other models on consumer GPUs and even some phones. Commenters highlight the privacy and “local‑first” benefits of this approach, but also flag real constraints: multi‑GB downloads stored in IndexedDB, uneven WebGPU support (especially on Firefox, Safari, and Linux), occasional loading bugs, and the risk that many sites might redundantly ship giant models. Some see this as a glimpse of a future where browsers or operating systems manage shared local models and standardized APIs, enabling offline assistants, coaching tools, and AI‑enhanced apps without relying on cloud providers.

Model size, storage, and browser limits

  • Models (e.g., Llama 3) are 4–5GB and downloaded on first use, then cached in IndexedDB; browser may later evict them.
  • Some users were surprised browsers can store multi‑GB blobs, having assumed ~400MB limits.
  • There’s interest in resources explaining modern browser quota behavior.
  • Concerns raised that if this pattern spreads, users may repeatedly download large models for many sites, filling disks.

WebGPU support and compatibility

  • App runs only with WebGPU (practically Chrome/Edge desktop; limited or experimental on Firefox, Linux, iOS Safari).
  • Several users report “Cannot find WebGPU” or “Cannot find adapter” errors; workarounds involve enabling flags or OS GPU settings, with mixed success.
  • Some devices (e.g., Pixel 6/6a, M1 Macs) run it surprisingly well; others hit driver/platform limitations.

Performance, UX, and bugs

  • Many report fast load and good token speed; some compare favorably to cloud GPT‑4 for everyday prompts on consumer GPUs.
  • Others argue browser-based LLMs still struggle with stability, lag, and hardware variability for “serious” UX.
  • A bug where resetting the chat mid-generation feeds partial output back into the model produces surreal “self‑hallucination”; users find it amusing and suggest a “mode” around it.
  • Requests for features: chat history, wider text layout, local document/knowledge integration (e.g., via retrieval systems).

Model quality and behavior

  • Llama 3 8B is seen as strong for its size but not flawless: some users experience poor contextualization (e.g., song-naming prompt derailing into repeated famous titles and incoherent backreferences).
  • Smaller models like Phi‑1.5 show odd “inner dialogue” and appear less usable; Phi‑3 and other 1–2B models are recommended as better small options.
  • Confusion over “web browsing”: models can convincingly hallucinate page summaries without actual network requests.

Local-first, privacy, and ecosystem ideas

  • Strong enthusiasm for private, local-first AI that avoids sending sensitive data to cloud providers.
  • Suggestions include:
    • browser- or OS-level model managers shared across sites,
    • extensions/APIs exposing a local LLM service to web apps,
    • deeper integration into browsers (e.g., standardized LLM APIs, permission prompts like camera/mic).