I burned all my tokens researching how to save tokens
Efforts to reduce large language model (LLM) costs are revealing an awkward tradeoff: many “token-saving” tricks add complexity, break context caching, or simply move waste around instead of eliminating it. Commenters compare strategies such as multi-model pipelines, frozen vs dynamic prompts, and local vs cloud deployments, with some arguing that cloud APIs remain cheaper and more capable than most on-prem setups, except where privacy or scale justify hardware investment. Underneath the technical details runs a broader tension over whether LLM-assisted code and products are genuinely valuable or mostly unshippable “slop,” and how to measure real productivity gains beyond anecdotal claims.
Token-saving strategies & research workflows
- Many describe similar experiences to the article: LLMs aren’t “ignorant” but lack discipline, burning tokens on repeated dead ends. Goal becomes reducing repeated dead ends, not exploration itself.
- Suggested pipeline: start with cheaper/smaller models for hypothesis generation and broad exploration, then pass distilled results to stronger models. Parallel low-cost hypotheses can outperform a single expensive call.
- Batch pricing and “overnight” jobs are proposed to cut costs when latency is acceptable.
- Some suggest mixing models from multiple vendors in early stages for diversity.
Context, caching, and prompt design
- Several report that clever token-saving tricks often break context-prefix caching, making things more expensive overall.
- Fixed, slightly larger prefixes plus occasional summarization can beat dynamic retrieval and pruning schemes.
- Letting the model manage compaction (e.g., via editor integrations) works surprisingly well for some; context can often just be allowed to fill, then summarized.
- Adaptive model routing by difficulty is discussed; problems include replaying entire histories when switching models and incompatibility of attention caches across models.
Local vs cloud models and economics
- Strong view that, for most people, cloud models are cheaper and better than local ones, unless you already have substantial hardware or strict privacy needs.
- Others argue local models make sense for organizations with many heavy users or existing compute, or as a hedge against vendor lock‑in and changing restrictions.
- A “90% local / 10% frontier” pattern is proposed, but identifying when to switch is seen as nontrivial.
Tools, agents, and avoiding repeated work
- Various tools are mentioned: memory/caching MCPs, skills files, and workflows that periodically update “rules/skills” based on past chats to stop redoing the same steps.
- Concern is raised that many agents and products keep re-solving the same issues; shared knowledgebases for agents are proposed to converge on reusable solutions.
Shipping, “slop,” and real-world value
- Heated debate over whether AI-assisted projects are genuinely useful or just “slop.”
- Some report shipping internal tools, migrations, games, and hardware more quickly with LLMs.
- Others question the value of shipping things users could now build themselves and criticize AI projects with bot-written documentation.
- Definitions of “shipped” (paid users vs personal tools vs OSS) are contested.
Hallucinations & reliability
- Skepticism that hallucinations can be eliminated by rules or pipelines; they’re seen as intrinsic to current models.
- Claims of “no hallucinations” are challenged as overstated.