Building a fully local LLM voice assistant to control my smart home

Original Article ↗ Hacker News Discussion ↗

A detailed home project to build a fully local, GLaDOS‑themed voice assistant for Home Assistant sparks broader interest in privacy‑preserving, cloud‑free smart homes. Commenters compare local models like Mixtral, Mistral, and TinyLlama to GPT‑4, weighing hardware needs, latency, quantization and GPU choices against the convenience of cloud APIs. Much of the exchange centers on how to safely let LLMs control real devices—using constrained grammars, function‑calling–style interfaces, and strict access controls—while Home Assistant’s roadmap hints at future built‑in support for local LLM automation and standardized APIs.

Project reception & similar builds

  • Many commenters are building comparable local voice assistants, often with Home Assistant as the hub.
  • Some use only local models; others prototype with OpenAI/Mistral APIs first, then aim to go local.
  • Several people share small wrappers or libraries to simplify function calling and integration with Python or Home Assistant.

Hardware, performance & quantization

  • Strong emphasis on using GPUs: main bottleneck is time-to-first-token, especially with large prompts including full home state.
  • Mid-range GPUs (e.g., 16 GB consumer cards) are seen as good VRAM-per-dollar and lower power draw than used datacenter cards, which can stress power supplies and UPSes.
  • 4-bit quantization (GPTQ, AWQ) is commonly used; people report ~17 tok/s on Mixtral-like models as “usable but not snappy.”
  • Some run 7B–20B models on 8–12 GB GPUs or even CPU-only for experimentation.

Model behavior, grammars & structured output

  • Multiple comments suggest using grammars (GBNF/BNF in llama.cpp) or libraries that constrain output to valid JSON, instead of relying on prompt-only formatting.
  • Discussion around whether Mixtral lacking system prompts makes it more vulnerable to prompt injection; some suggest fine-tuned variants or other models with better “system” support.
  • Debate over capability of ~7B models: some find them near GPT‑4 for narrow tasks; others call them unreliable for complex, structured automation.

Home Assistant integration & future direction

  • The Home Assistant project intends to ship LLM-based functionality, but wants:
    • A richer, standardized local-LLM API (beyond “just copy OpenAI”).
    • Robust function-calling or constrained grammars so JSON actions are always directly safe to execute.
  • Ideas: rulebooks in natural language for household behavior, AI-suggested automations from history, and one-click switching among local models via add-ons.
  • Concerns raised about hardware requirements; others note HA is modular and LLMs/STT/TTS can run on separate, more powerful machines.

Safety, security & networking

  • Several commenters worry about LLMs controlling physical devices (ovens, locks, doors). Suggested mitigations:
    • Hard-coded safety checks on outputs.
    • Limiting which services/entities the LLM may call.
    • ACL/RBAC-like controls, sometimes via undocumented HA APIs.
  • Some fear malicious or “sleeper” models; others argue the real risk is general IoT exposure, not LLMs specifically.
  • Exposing Home Assistant directly to the internet is controversial; a few do it behind WAFs/VLANs, but most prefer VPN/WireGuard.

Voice pipeline & UX

  • Latency is a recurring concern: 8+ seconds from speech to first reply is widely considered marginal.
  • Suggestions:
    • Front-end grammar/intent rules for simple commands.
    • Caching frequent utterances and even TTS audio.
    • Early “one moment” responses and response streaming to TTS.
  • Wake-word and mic quality remain practical pain points; ESP32-S3 devices with wake-word models and I2S mics are popular experiments.