If you don't run your own LLM proxy, how do you know whats going on?
Growing reliance on AI coding agents and LLM “harnesses” is pushing developers to seek deeper observability and control over what these tools are doing with their code, data, and API keys. Many advocate running self-hosted LLM proxies or using OpenTelemetry-based tracing to log prompts, tool calls, costs, and guardrails, while others argue this adds fragile complexity compared to vendor-managed subscriptions. Underneath is a broader concern about security, vendor lock-in, opaque data usage, and the long-term economics of subsidized LLM subscriptions versus pay-per-token APIs and local models.
Why run an LLM proxy or observability layer?
- Many want “full observability” into agents: prompts, tool calls, responses, token usage, and reasoning.
- Proxies or tracing let people:
- Track costs per project/org.
- Debug complex, multi-agent workflows.
- Enforce guardrails (e.g., limited tools, filtered web results, virtual API keys).
- Avoid leaking provider keys into sandboxes or untrusted environments.
- Some users archive all AI chats into personal “corpora” to search and reuse later.
Tools and approaches mentioned
- Proxies / gateways: LiteLLM, Bifrost, GoModel, Aperture, llama-dash, custom gateways, local “AI ops” dashboards.
- Non-proxy observability: OpenTelemetry (OTEL) traces from Codex/Claude Code and plugins for other harnesses; MLflow-based tracing; various dashboards and history viewers.
- Some prefer simple MITM proxies or just scraping local JSONL/chat history over full-blown gateways.
Subscription vs API pricing debate
- Several report maxing out high-end subscriptions quickly, with equivalent API costs being 10–30x higher. Others say even a $25/month sub is hard to exhaust.
- General view: personal subscriptions are subsidized “loss leaders,” while enterprises must pay API rates.
- Disagreement on whether subsidies will end soon vs. remain as long-term acquisition tools.
- Some argue per-token API use can be cheaper and more “honest” if you tightly control demand; others prefer flat-rate subs to avoid cost anxiety and over-optimization.
Security, privacy, and compliance concerns
- Proxies and gateways are used to hide provider keys, centralize policy, and keep credentials out of agents.
- EU posters warn that chat logs often include personal or sensitive data and must be treated as regulated “personal data,” not just code.
- Some stress sandboxing harnesses in VMs and monitoring all outbound network traffic, since OTEL only shows what tools choose to report.
- Enterprise plans often required for strong data-use guarantees, but these are API-priced and expensive.
Skepticism and downsides
- Concerns about proxies and LLM gateways as “buggy, vibecoded messes” and new single points of failure.
- Others feel this level of setup is “setup-theatre” for hobbyists; many are content with default subscriptions and minimal monitoring.
- Some argue OTEL-based tracing is safer and less intrusive than inserting a proxy.