Show HN: Smart model routing directly in Claude, Codex and Cursor

Original Article ↗ Hacker News Discussion ↗

Smarter model routing tools are emerging to cut the cost of AI-assisted coding by automatically choosing between expensive “frontier” models and cheaper or open-source alternatives. Commenters probe whether such routers can truly save money once cache invalidation, long-running agent sessions, and small-model failure modes are accounted for, and note that many IDE-style harnesses already make their own routing decisions. Key concerns include cache-aware design, data privacy when using proxies, the difficulty of keeping routing policies up to date, and the lack of published benchmarks to validate claimed cost and productivity gains.

Cost, Caching, and Token Savings

  • Many commenters worry that multi-model routing increases cache misses and thus cost, especially with short cache TTLs and long coding sessions.
  • The router claims to be “cache-aware”: once a model is in use, switching requires a higher threshold so that cache loss is offset by cost or quality gains.
  • The authors report ~40% internal cost savings, arguing that:
    • Small models can fully handle many requests.
    • Using a small model for part of a task plus one cache miss is often cheaper than using a frontier model end‑to‑end.
    • Subagents with fresh contexts are natural places to route to cheaper models.
  • Some argue that beyond one planner + one executor model, extra models mostly add cache penalties and complexity.

Routing Logic, Reliability, and Use Cases

  • The router is trained on real agent traces; reward signals reflect whether a chosen model completes tasks successfully.
  • It uses embeddings of prompt + context to cluster similar tasks and adapt over time; larger models can “rescue” smaller ones that get stuck.
  • Concerns: small models may loop or fail more; ambiguous tasks are hard to route correctly; many users tune prompts to specific models, so automatic routing may misalign with their expectations.
  • Several see this as more suited to average users or agentic coding setups than to highly controlled, pre‑evaluated production flows.

Integration, Subscriptions, and Privacy

  • It can use existing Claude/Codex subscriptions (subsidized usage) and fall back to API billing; also integrates with tools like OpenCode and can route to Gemini.
  • Some worry about data privacy when using any proxy. The project can be self‑hosted; many customers reportedly still choose the hosted option.

Comparisons and Alternatives

  • Compared to other routers (e.g., vLLM Semantic Router, Sakana Fugu, Cursor “auto”), this one emphasizes agentic coding and cache awareness.
  • Some note that coding harnesses and IDEs already do their own model-aware routing and question whether a proxy-layer router breaks that control loop.

Evidence and Open Questions

  • Commenters request public benchmarks (e.g., coding benches, RouterArena) and details on switching frequency.
  • The maintainers say internal metrics show unchanged code quality and higher velocity, but external evals are not yet linked.