Launch HN: RunAnywhere (YC W26) – Faster AI Inference on Apple Silicon

Original Article ↗ Hacker News Discussion ↗

A new Apple Silicon–focused toolchain, RunAnywhere’s MetalRT engine and open-source RCLI client, promises significantly faster on-device LLM, speech-to-text, text-to-speech, and local RAG pipelines, aiming to make voice-driven assistants and private document querying feel as responsive as cloud services. Commenters probe real-world performance, model size limitations, installation quirks, licensing (MIT CLI atop a proprietary engine), and the tradeoff between small, fast models and reliable tool calling, while noting planned support for larger models, more quantization formats, and mobile. The launch also revives criticism of the company’s earlier GitHub-focused email outreach and raises questions about Hacker News moderation and voting patterns, highlighting how product excitement is tempered by concerns about trust and ethics.

What RunAnywhere / RCLI Provides

  • Company builds MetalRT, a proprietary inference engine for Apple Silicon, plus RCLI, an open-source CLI demo.
  • RCLI wires together local speech-to-text (STT), LLM, text-to-speech (TTS), local RAG, and macOS actions into a voice assistant / TUI.
  • Emphasis on fully local processing and no telemetry by default.

Performance & Model Choices

  • Benchmarks claim MetalRT is modestly faster than competing Apple-Silicon engines (e.g., MLX, uzu) for 0.6B–4B models and much faster for STT/TTS.
  • Some see those small models as “toy-sized” and ask for benchmarks on 7B–70B+ models; founders say larger models are on the roadmap.
  • Commenters note unified memory makes Apple Silicon attractive for very large models; current MetalRT support is focused on latency-sensitive voice pipelines.

Use Cases & Feature Requests

  • Suggested uses: always-on dictation, virtual audio devices for real-time transcription in video calls, and on-device RAG over sensitive documents.
  • RCLI supports local RAG with fast hybrid retrieval, and text-only mode (no TTS).
  • Requests include better quantization formats (e.g., unsloth), richer TTS voices, diarization, Linux support, and SDK access for third-party apps.

Quality & Limitations

  • Tool-calling with small models is unreliable: commands may be “acknowledged” verbally without the correct macOS action firing.
  • Team acknowledges this as a core unsolved problem for sub-4B on-device models and plans verification layers and larger models.
  • Default TTS quality is criticized as dated; better models (e.g., Kokoro) are available but not default.

Installation & Platform Support

  • Some users report segfaults and Homebrew install issues.
  • Install script silently installing Homebrew is widely criticized; maintainers agree to change it.
  • MetalRT currently targets M3/M4; M1/M2 fall back to llama.cpp. Mobile support and other edge devices are planned.

Licensing & Openness

  • RCLI is MIT-licensed; MetalRT and many models are proprietary.
  • Some see a closed inference engine as “reinventing the wheel” versus CoreML/MLX; maintainers argue specialization yields higher performance and unified STT/LLM/TTS support.

Security & Trust Concerns

  • A web demo leaked third-party API keys; initial “bait” response is criticized as flippant, later walked back with an apology and promise to fix.
  • Separate prior controversy: company scraped GitHub data and used lookalike domains for cold email campaigns. Several commenters say this permanently damaged their trust.

HN Voting & Moderation Discussion

  • Some users suspect vote manipulation due to fast rise and comment ordering.
  • Moderation explains: YC “Launch HN” posts receive special placement; off-topic comments (e.g., about past behavior rather than product) are downweighted but not removed.