WhisperSpeech – An open source text-to-speech system built by inverting Whisper

Original Article ↗ Hacker News Discussion ↗

An open-source project called WhisperSpeech is using OpenAI’s Whisper speech recognition model in reverse to create high-quality text-to-speech, aiming to close the gap between proprietary systems like ElevenLabs or OpenAI’s own TTS and what can be self-hosted. Commenters explore its current strengths (natural prosody, voice cloning, real-time performance on GPUs, fully local/offline use) alongside limitations such as language coverage, compute requirements, and the practical challenges of datasets, tooling, and legal uncertainty around training data. Comparisons with other TTS solutions (Piper, Mimic, Azure, Google, EmotiVoice, Tortoise) highlight trade-offs between quality, cost, latency, and openness, and illustrate how hard it still is to turn rapid ML advances into robust end-user products.

Training approach & architecture

  • Uses Whisper’s multilingual ASR encoder as a “semantic” front-end, so training only needs speech audio; transcripts are auto-generated by Whisper.
  • Architecture is described as: Whisper encoder → semantic tokens → acoustic tokens (from Encodec) → Vocos vocoder.
  • This sidesteps proprietary semantic encoders (e.g., in VALL-E/SPEAR-TTS) and enables multilingual extension if data exists.

Features, voice cloning & tunability

  • Supports voice cloning from a reference audio file (example in the Colab notebook, not yet well documented in the README).
  • Voice identity is relatively tunable; prosody and emotion are still hard to control and are an active research area.
  • Can convert existing speech to different voices via Whisper-derived tokens; accent transfer is speculated but not yet tested.

Performance, hardware & deployment

  • Runs locally; no cloud, no tracking. Weights are downloaded once from Hugging Face.
  • On an RTX 4090, reported ~12× faster than real time; expected real-time performance for a “voice-bot” on modern NVIDIA GPUs.
  • Current implementation is CUDA/PyTorch only; ports to MLX and whisper.cpp/llama.cpp-style frameworks are desired.
  • Inference models use ~3 GB VRAM in FP16 (2.3 GB FP32, suggesting room for optimization).
  • Full training from scratch: ~8 hours on 96 A100 GPUs for the main TTS model; smaller models are faster. Fine-tuning is possible but not yet user-friendly.

Language support & datasets

  • Currently supports English and Polish; multilingual expansion is planned.
  • Relies on high-quality audiobooks (e.g., public domain/CC sources like LibriVox, WolneLektury), plus smaller sets with varied emotions/prosody.
  • Community suggests Mandarin corpora and even precomputed forced alignments; overall bottleneck is clean, legally usable data.

Quality comparisons

  • Many commenters find WhisperSpeech among the best open-source TTS; some say Tortoise is slightly better but far slower.
  • Compared to Mimic 3/Piper and EmotiVoice, several note WhisperSpeech sounds more natural; EmotiVoice English is described as somewhat non-native.
  • Commercially, ElevenLabs is viewed as top quality but very expensive; OpenAI’s TTS is nearly as good and far cheaper, Azure Neural/Google TTS are lower quality but have generous free tiers.

Licensing & legal concerns

  • Project emphasizes using properly licensed (PD, CC-BY/SA) speech data and fully open-source code for safer commercial use.
  • Debate arises over the phrase “properly licensed” since Whisper’s own training data is unclear; distinction is made between:
    • Using Whisper (MIT-licensed, non-generative) outputs as input features.
    • Potentially problematic training of generative models on unlicensed data.
  • Acknowledgment that future legal changes could alter the risk landscape; documentation on dataset licenses is being improved.

Ecosystem, tooling & applications

  • Interest in integrating WhisperSpeech into:
    • VR social platforms for text-based users.
    • Desktop tutorial apps.
    • Chat/assistant systems needing many distinct voices.
  • Broader discussion notes:
    • Rapid model progress vs. slow, painful productization.
    • Tooling pain: GPU dependencies, poor install scripts, and integration hurdles.
    • Examples of live captioning/translation systems built on Whisper; WhisperLive is mentioned as a practical project.