Tokens too cheap to meter
Claims that AI “tokens” will soon be so cheap they’re effectively unmetered provoke sharp debate over how fast inference costs can really fall and whether hardware and energy constraints will slow that trajectory. Commenters weigh the possibility of frontier‑level models running locally on consumer devices against concerns about RAM and GPU supply, data‑center overbuilding, and environmental impact. Many agree that inference is likely to become a low‑margin commodity service, but question whether today’s massive capital investments in models and infrastructure can ever be justified by future profits.
Hardware, Local LLMs, and Memory
- Debate over whether frontier-quality models will run locally on commodity hardware within 3–6 years.
- Some argue historic trends in compute/RAM per dollar are still roughly exponential and that prices “crash” in sawtooth cycles when capacity overshoots demand.
- Others note current supply constraints, slow fab buildout, and doubt sub-2030 normalization of RAM/GPU prices.
- Several see ASICs and model distillation (plus techniques like quantization, MoE, speculative decoding) as key to making local and cheap inference viable.
Data Centers, Demand, and Possible Bubble
- One camp views the current data center buildout as a bubble: efficiency gains and model improvements could outpace demand, leaving stranded capacity.
- Others invoke Jevons’ paradox and near-infinite demand for “intelligence,” arguing cheaper tokens will drive more usage, not less.
- Some stress Jevons is not a law; overbuilds in other industries show supply can outstrip economically meaningful demand.
Economics, Profitability, and Business Models
- Strong disagreement on whether inference is already a “cash cow.”
- Critics point out labs are deeply negative on free cash flow, rely on adjusted EBITDA, and face relentless capex to stay SOTA.
- There’s concern current prices are subsidized and not reflective of full costs (training, hardware, debt).
- Others counter that inference-only providers on open-weight models appear profitable, and that renting compute/inference will settle into low-margin commodity business similar to cloud.
- Question raised: can any one firm recoup trillion‑scale investments, or will later, leaner entrants capture commodity inference profits?
Token Costs vs Tools (grep, compilers, etc.)
- Thread challenges the article’s claim that LLM calls will soon be cheaper than simple tools like
grep. - Many see 4–5 orders of magnitude cost gap as huge and likely to persist; more plausible is an asymptotic approach, not crossing.
- Some argue LLMs can be cheaper for approximate, higher-level tasks or when attention/UX cost dominates, but baseline deterministic tools remain cheaper per operation.
Energy and Environment
- Concern that large-scale AI compute could be environmentally “bad to disastrous.”
- Others respond that strong economic pressure exists to reduce energy per token; ASICs, distillation, and embedding models in devices could drastically lower energy use.
Historical Analogies and Hype
- Frequent comparison to “too cheap to meter” nuclear promises and the dot-com era: useful technology, but current capital spend and valuations may be unsustainable.
- Mixed sentiment: optimism about long-term ubiquity of AI vs skepticism about near-term projections, charts, and hype cycles.