Show HN: WhisperFusion – Low-latency conversations with an AI chatbot

Original Article ↗ Hacker News Discussion ↗

Low‑latency, fully local voice assistants like WhisperFusion are emerging as a credible alternative to cloud-based systems such as Siri and Alexa, combining streaming speech recognition (WhisperLive), a compact LLM, and text‑to‑speech for near real‑time back‑and‑forth. Commenters explore technical hurdles—especially Whisper’s 30‑second chunking, streaming latency, and end‑of‑turn detection—as well as UX needs like natural interruption, variable response timing, and conversational cues. There is strong interest in privacy‑preserving, home‑automation and coding companions, but also concern about deployment complexity, GPU requirements, and the limits of repurposing models not originally designed for live audio.

Overall Reaction and Use Cases

  • Many see WhisperFusion as a strong step toward what Siri/Alexa “should have been”: low‑latency, conversational, and useful for home automation, hands‑free interaction, and “super smart friend” scenarios.
  • People imagine use cases like walking and chatting, ordering food by voice, interacting with devices around the house, and pair programming/learning with an AI that sees the screen.

Local, Private Voice Assistants

  • Strong interest in everything running locally to avoid sending continuous audio streams to remote APIs.
  • Some argue users will increasingly demand local‑only AI, especially when it has full screen and microphone access, though others think most people won’t care until directly harmed by leaks.

Latency, Streaming ASR, and Model Choices

  • Project uses WhisperLive (ASR), WhisperSpeech (TTS), and small LLMs (Phi‑2, Mistral) optimized with TensorRT for low latency.
  • Whisper’s 30‑second chunk design is criticized as ill‑suited for sub‑second streaming; some call the workaround (padding, continuous reprocessing) inefficient and potentially high‑latency.
  • There are calls for concrete WER and latency metrics, ideally graphs; current documentation is seen as lacking.
  • Alternatives and future directions discussed: W2v‑BERT 2.0, recurrent / Mamba‑like models, Kaldi/sherpa and Parakeet streaming models, chunked attention architectures.
  • It’s unclear how WhisperLive’s real‑world latency/WER compares to specialized streaming ASR research that reportedly achieves ~1s latency with minimal WER degradation.

Conversation Dynamics and UX

  • Major pain points: premature turn‑taking on short pauses, inability to interrupt, fixed speaking speed, and text‑style responses in voice mode.
  • Desired features: explicit interruption (“hang on”), explicit cues before response (“What do you think?”, “Over”), content‑based end‑of‑turn detection, and prediction of when to speak.
  • Some propose predicting turn‑taking directly from audio and doing continuous “intention” and conversation‑style modeling to make interactions feel human.

Vision and Context Integration

  • Interest in a desktop/mobile assistant that can access screen context via screenshots, accessibility APIs, or multimodal models, while managing latency and hallucinations.
  • Longer‑term ideas include in‑home vision to track objects (“where are my keys?”) and persistent “artificial attention” that remembers everything on screen.

Implementation, Deployment, and Hardware

  • Praise for the project but frustration with Python packaging complexity, TensorRT’s “examples”‑centric distribution, and Hugging Face’s custom download tooling.
  • Users request simple installers (.deb/.rpm/.dmg) or at least a single Docker command; maintainers plan to simplify the Docker-based demo.
  • Tested on RTX 3090/4090; Jetson support and CoreML backends are desired but currently “in progress” or unclear.