StyleTTS2 – open-source Eleven-Labs-quality Text To Speech

Original Article ↗ Hacker News Discussion ↗

An open-source text-to-speech model called StyleTTS2 is drawing attention for delivering fast, high-quality synthetic voices that some users see as approaching commercial systems like ElevenLabs, especially when run on consumer GPUs. Commenters explore practical aspects such as hardware requirements, Python/CUDA setup, language support, and how well the tool handles longer-form audio and voice cloning, with many noting that cloning quality still lags behind leading paid services. The thread also surfaces licensing ambiguities around the model weights and broader concerns about misuse for scams, alongside enthusiasm for applications in local AI assistants, games, audiobooks, and conversational agents.

Licensing and “open source” status

  • Code is MIT-licensed, but pre-trained weights come with extra conditions: users must disclose AI synthesis unless they have permission from the voice owner.
  • Several commenters argue this is not “pure” MIT and makes commercial use risky or ambiguous; others note you can avoid restrictions by training your own models.
  • Some people are bothered that the repo/announcement doesn’t clearly separate code vs. model licensing.

Quality vs. ElevenLabs and other TTS

  • Many say this is the best open-source TTS they’ve heard: natural prosody, fast, and sometimes “better than ground truth” in demos.
  • Others find a noticeable metallic tone and say it’s still below ElevenLabs and OpenAI TTS, especially for long-form or voice cloning.
  • Some think the HN title (“Eleven Labs quality”) is exaggerated and editorializing.

Voice cloning performance

  • Zero-shot voice cloning is widely reported as weak: pitch and cadence may transfer but accent/timbre often don’t, sometimes producing generic or British-sounding voices.
  • Compared unfavorably to XTTSv2 and ElevenLabs; authors’ own paper reportedly acknowledges cloning isn’t strong yet.
  • Explanations focus on training data scale (hundreds vs. tens/hundreds of thousands of hours).

Hardware, performance, and setup

  • Models ~700MB; inference can be faster than real-time on GPUs (e.g., 15–95x RT on 4090; ~2x RT on older CPU laptop).
  • Works on CPU-only but much slower; Raspberry Pi 4 is “minutes per utterance.” Alternative lightweight TTS (e.g., Piper) suggested for Pi.
  • Some report installation is fiddly (dependencies, CUDA versions, venv proliferation). Others share working step-by-step setups, recommend mamba/conda or Docker.

Use cases and integrations

  • Strong interest in:
    • Local conversational assistants combining StyleTTS2 + LLMs + Whisper.
    • E-book / article → audiobook pipelines, including self-hosted services.
    • Game NPC dialogue, dynamic narration, and modding (e.g., Skyrim, golf sims).
  • One user built a fully local Windows voice chatbot (Whisper + Mistral + StyleTTS2) with very low latency and interruption support.

Language support

  • Core pretrained TTS models are English.
  • Auxiliary components (ASR, pitch extractor, phoneme BERT) have varying multilingual support, but full non-English TTS currently requires extra training.

Ethics, safety, and jobs

  • Some warn of scams and elder fraud via voice cloning, advocating only service-based access and watermarking.
  • Others argue abuse is already happening with existing tools and that policy/penalties, not blocking models, is the right response.
  • Debate over impact on voice actors: some see displacement of “commodity” work; others expect new markets for licensed voices and expanded use of performers.