Price per 1M tokens is meaningless

Original Article ↗ Hacker News Discussion ↗

Limits of “price per 1M tokens”

  • Many see price-per-token as misleading on its own: models differ in verbosity, required “thinking”/effort settings, tool use, and caching behavior, all of which change real cost.
  • Subscriptions distort pricing further: flat monthly plans can give massive discounts vs on-demand tokens, but usage limits are often opaque.
  • Others argue token pricing is still a useful base metric, analogous to fuel price or hourly wage: incomplete, but you need some standard unit.

Cost per task and benchmark relevance

  • Several commenters favor “cost per benchmark task” or “cost per prompt” as better measures of real-world cost.
  • Caveat: benchmarks must match your workload.
    • If your tasks are harder than the benchmark, a “cheap per task” model that fails your task is useless.
    • If your tasks are much easier, you might overestimate how expensive a stronger model is.
  • Some suggest routing tasks to different models by complexity and using a harness to abstract model choice.

Speed, UX, and verbosity

  • Speed and latency matter as much as raw token cost, especially for interactive tasks like commit message generation.
  • Verbose models can be slower and more expensive overall, even if they have higher tokens-per-second or lower per-token price.
  • For local LLMs, prompt-processing speed can dominate in multi-turn/agentic workflows.

Caching, tool use, and orchestrators

  • Caching can dramatically cut effective costs (some reports of ~0.1x or very high cache hit rates), but efficiency varies widely by provider.
  • Tool calls and agent frameworks often re-send large contexts, multiplying token usage; poor context management can bloat costs.
  • Some describe Kanban-style orchestrators and agent pipelines where smaller/cheaper models do most worker tasks, with larger models only for planning.

Open vs frontier models and deployment choices

  • Disagreement over whether open models are “good enough” for end-to-end agentic workflows; some claim frontier models are essential, others report strong results with open models at a fraction of the cost.
  • Debate over in-house vs cloud: one camp urges buying GPUs and optimizing locally to explore use-cases at predictable cost; another notes that many lack hardware/skills and rely on hosted open-weight models.

Broader skepticism and “AI bubble” concerns

  • Some emphasize that LLMs are just text predictors and that real value comes from the harness and workflow design, not the raw model.
  • There is concern about over-optimizing for gamable metrics and a belief that efficiency (cost per task, tasks per kWh) will matter most if/when an AI bubble deflates.