Nvidia's Chat with RTX is an AI chatbot that runs locally on your PC

Original Article ↗ Hacker News Discussion ↗

Nvidia’s new “Chat with RTX” tech demo offers a local AI chatbot that runs Mistral or Llama 2 on Windows PCs with recent RTX 30/40-series GPUs, indexing users’ documents and web content via RAG so they can query their own data. Commenters see it as both a showcase for Nvidia’s TensorRT-LLM acceleration and a bid to lock AI workloads to newer hardware, sparking complaints about artificial GPU and VRAM requirements and comparisons to existing open-source tools that are harder for non-experts to set up. There is also debate over privacy, censorship, and whether brand-backed, one-click installers will bring local LLMs to a much wider audience than current DIY solutions.

Hardware Requirements & Limitations

  • Official requirements: Windows 11, 16GB RAM, RTX 30/40 series or RTX Ampere/Ada GPU with ≥8GB VRAM, recent driver.
  • Many complain that 20‑series RTX (e.g., 2080 Ti with 11GB) are excluded and see this as an artificial, branding-driven limitation.
  • Some argue it’s due to missing/bad support for lower-precision formats (e.g., bf16) and first-gen tensor cores; others note TensorRT-LLM itself lists Turing as supported, so the restriction feels arbitrary.
  • 8GB VRAM minimum also excludes cards like the newer 6GB RTX 3050, which users find frustrating.

Models, Performance & Tech Stack

  • Installer is ~35GB and includes LLaMA 13B and Mistral/Ministral 7B in int4 quantization; on 8GB cards it appears to default to Mistral 7B.
  • Backend is TensorRT-LLM plus a Windows RAG wrapper; UI is a thin Gradio-based front end with a bundled Conda environment.
  • Reported performance is very fast (e.g., “faster than I can read,” hundreds of tokens/s on high-end cards), at the cost of low variability and limited conversational context.
  • Several commenters note TensorRT-LLM is one of the faster inference engines versus alternatives like llama.cpp, though harder to set up manually.

Purpose, Audience & Value vs Existing Tools

  • Some struggle to see the point versus tools like text-generation-webui, Ollama (on non-Windows), LM Studio, etc.
  • Others argue the value is:
    • Official Nvidia branding and support.
    • One-click installer and simple UI for non‑expert Windows users.
    • Automatic model choice based on VRAM.
  • Comparison: open-source stacks are seen as more flexible but intimidating (GitHub, terminal scripts, jargon-heavy UIs).

RAG and “Chat with Your Files”

  • Distinguishing feature highlighted: built-in RAG over local data (documents, YouTube URLs).
  • Some note RAG quality is mixed: correct answers but sometimes wrong file citations.
  • People see strong appeal in a local personal assistant indexing all their activity and documents, but acknowledge this is not fully realized yet.

Local vs Cloud, Privacy & Censorship

  • Users like that it’s local and potentially less censored than cloud AIs, but worry Nvidia might still collect data.
  • Source code (minus installer) is available for inspection; some recommend monitoring/blocking network traffic if concerned.
  • Legal/PR risk is cited as reason Nvidia will still include content guardrails.

Market & Ecosystem

  • Seen as a tech demo for TensorRT-LLM and a way for Nvidia to push RTX as an “AI” brand, not just ray tracing.
  • Some lament lack of Linux version; others point to the underlying TensorRT-LLM repo for Linux users.