Computer Use is 45x more expensive than structured APIs

Original Article ↗ Hacker News Discussion ↗

Benchmarking shows that LLM-based “computer use” via browser/vision agents can be roughly 45× more expensive and far slower than calling well-structured APIs for the same tasks. Commenters largely agree that GUI-driven automation should be a last resort, useful mainly for legacy or locked-down systems without APIs, while real-world agentic products will need deterministic, schemaed interfaces (MCP, REST, CLI, accessibility APIs) to be affordable and reliable. The thread also explores how this cost gap could reshape OS and app design, revive interest in programmability and accessibility, and encourage hybrid approaches that map UIs into reusable workflows or pseudo-APIs.

Overall reaction to the benchmark

  • Many find the result unsurprising: UIs built for humans are inefficient for machines; APIs are the “obvious” better interface.
  • Others stress the value is in quantifying the cost gap and showing wall‑clock differences (seconds vs ~17 minutes).
  • Some argue it’s “only” 45x, and expected the gap to be even larger.

When “computer use” is still needed

  • Key use cases: legacy / proprietary / locked‑down apps (EHRs, hotel PMS, archaic SaaS) and sites with no or closed APIs.
  • Examples include medical billing systems where vendors block API access, forcing computer vision + OCR to navigate screens.
  • Some note it’s useful for GUI debugging or end‑to‑end testing where UI behavior (not just API) must be validated.

Critiques of the experiment and models

  • Commenters note vision agents failed on basics like scrolling, inflating token use; some call this a model or prompt‑design problem.
  • Several say the comparison is based on a single workflow, so more like a case study than a true benchmark.
  • There’s interest in testing other browser/agent tools (Playwright, agent‑browser, dev‑browser, hybrid DOM/vision approaches).

OS, UI, and “agent‑first” design

  • One camp argues an “agentic world” needs OSes and apps with first‑class APIs for all functionality, potentially via MCP‑like surfaces, DBus, accessibility APIs, etc.
  • Others push back: incentives for consumer apps (ads, dark patterns, data hoarding) work against exposing clean automation APIs.
  • Historical analogues are mentioned (AppleScript, OLE Automation, Unix shells, REST/HATEOAS); many note they were underused or abandoned.

Cost, latency, and tokens

  • Strong consensus: generic computer/vision use is currently too slow and token‑hungry for most real products; APIs and CLIs are cheaper, faster, and more reliable.
  • Some report browser/computer use quickly exhausting plan limits; others experiment with “code once, reuse many times” (e.g., generating Playwright scripts or reusable workflows) to amortize cost.
  • A minority argue computer use can be competitive when APIs are verbose or nonexistent and when screenshots are small.

Trust, safety, and practicality of agents

  • Many are reluctant to let agents handle high‑stakes, sensitive tasks (taxes, background checks, company workflows) due to hallucinations and accountability concerns.
  • Agents are often likened to interns that need close supervision, not autonomous “real” agents.

Emerging patterns and meta‑effects

  • There’s a visible “mean reversion” toward deterministic, structured interfaces (APIs, schemas, CLIs).
  • AI is unexpectedly driving better documentation, accessibility, and structured design, since these directly improve agent performance for both humans and machines.