Launch HN: Momentic (YC W24) – Agents that catch bugs by using your app

Original Article ↗ Hacker News Discussion ↗

AI-powered QA agents that actively use web and mobile apps to hunt for bugs are drawing both interest and skepticism. Commenters probe Momentic’s claims of higher bug recall versus generic LLM agents, questioning cost per bug, vendor lock‑in, determinism, and how quickly custom “harnesses” will be commoditized as frontier models improve. Many see value in automated exploratory testing and natural-language E2E tests, but raise concerns about closed-source infrastructure, onboarding UX, and whether teams would rather adopt open, self-hosted tools or plug in their existing LLM subscriptions.

Pricing, Tokens, and “Bring Your Own” Models

  • Multiple commenters dislike yet another SaaS that charges for LLM usage instead of letting users plug in their own API keys / subscriptions.
  • “Bring your own key/agent” is suggested as ideal so existing Claude/GPT subscriptions can be reused.
  • Founders say they experimented with BYOK/BYOA but off‑the‑shelf models underperformed; custom-tuned agents are core to the current product, though BYOK is not ruled out long term.

Moat, Competition, and Frontier Models

  • Several argue there’s little moat: frontier models and generic agents will soon match these capabilities, making a separate SaaS fragile.
  • Others counter that architecture, harnesses, durable context, and infra can remain valuable even as base models improve.
  • Open-source tools and internal DIY harnesses are cited as credible alternatives, especially for cost‑sensitive or self‑hosting teams.

Positioning vs Existing QA and Coding Agents

  • Product is framed as fully automated exploratory testing / UAT that runs on live apps (web, mobile, soon desktop), focusing on bug recall.
  • There is also a regression product using natural-language tests stored in the repo; lock‑in exists but is somewhat mitigated by plain English formats.
  • Tool is meant for devs, QA, PMs, and leadership; it integrates with other coding agents via CLI rather than replacing them.

Benchmarks, Metrics, and “Cost per Bug” Confusion

  • The benchmark claims higher bug recall than Codex, Claude Code, Devin, etc., but at higher cost per bug than Codex.
  • Several readers find the writeup confusing and interpret it as a better ad for Codex; others argue recall is more important than cost or precision in QA.
  • There’s interest in additional metrics: false negatives (silent passes), severity of bugs, “unsure” outcomes, human baselines, and neutral / third‑party benchmarks.
  • Methodology questions include whether bugs were planted, how human review was done, and how non‑reproducible findings are treated.

UX, Onboarding, and Privacy Concerns

  • Users report confusing UI states (bug counts vs “no bugs yet”), missing/late “stop testing” controls, lack of simple exports, and friction around auth setup.
  • Auto‑creating Slack connections from emails without clear consent is criticized.

Mobile, Infra, and Technical Details (Partly Unclear)

  • Questions raised about: real devices vs simulators, support for specific iOS versions (some quickly added), handling retry loops, access to network traffic, and depth of mobile bug visibility; many of these details remain unclear in the thread.