Lessons after a Half-billion GPT Tokens

LangChain, RAG, and Abstractions

• Many find LangChain bloated and “premature abstraction”; they prefer treating LLMs as flaky HTTP APIs with thin custom wrappers.
• RAG performance is described as highly data‑dependent: good on FAQ‑like or short, uniform chunks; poor on messy B2B PDFs/Word docs, rare entities, and keyword‑driven queries.
• Some view vector search as weak for named entities and IDs; others suggest fine‑tuning embedding models on domain data rather than tuning the LLM.
• Several note they started with RAG/LLM stacks but later found classic search or databases alone were more reliable for some tasks.

Prompting, Null Hypothesis, and JSON/Structure

• Strong recurring theme: simpler prompts often outperform long, detailed instruction lists, especially with GPT‑4; GPT‑3.5 often needs more explicit constraints.
• LLMs struggle to return “nothing” or “no result”; people work around this by:
  • Forcing explicit sentinel tokens or default strings (“NOPE”, special indexes like 999).
  • Splitting tasks into “is there anything?” then “list it” steps.
• Many run into issues with list length and JSON arrays; workarounds include indexed keys (item1, item2) or enforcing IDs. Some argue JSON mode or function calling help; others say well‑designed prompts already suffice.

Model Choice: GPT‑4 vs Claude vs Local

• Sharp disagreement on model quality: some see GPT‑4 as clearly superior and more stubbornly correct; others find Claude 3 Opus better for code explanations, longer outputs, and following style.
• Claude is praised for cheaper strong variants, but also criticized for hallucinated APIs and weaker grounding than GPT‑4.
• Local/open models (e.g., modern Mistral‑class) are suggested as “good enough” for many extraction/classification tasks, with benefits in determinism and reduced vendor risk.

What LLMs Are Good (and Bad) At

• Widely agreed strengths: summarization, knowledge management, document extraction, basic classification, data labeling, and tier‑one customer support.
• Weaknesses: reliable reasoning, complex multi‑step “agentic” behavior, strict null handling, factual lookup at scale, and deep table extraction from PDFs (often offloaded to specialized OCR/EDA tools).
• Some argue LLMs are transformative “force multipliers”; others call them over‑hyped word predictors, useful mostly as brainstorming/boilerplate tools.

Evaluation, Testing, and Uncertainty

• Testing prompts is described as slow and tedious; people run batches of prompts repeatedly, sometimes on smaller or local models for faster iteration.
• Ideas for uncertainty: logprobs, self‑evaluation steps, running multiple samples, or explicit probability scores—but many caution these are still token‑level proxies, not true introspective confidence.
• Debates arise over whether models “reason” or merely “pretend to reason”; no consensus, but several note that RLHF and feedback can push behavior toward more useful patterns without changing the underlying token‑prediction nature.

Product Impact, UX, and Cost/Sustainability

• Some teams process billions of tokens/month and see clear commercial value, especially when AI is embedded so users barely notice it.
• Others want more concrete evidence of ROI, not just technical anecdotes.
• UX is considered as important as model choice; “generative UI” and careful interaction design can matter more than raw LLM capability.
• Cost sensitivity is growing: large models are seen as expensive relative to output; expectation is prices will fall, but some argue interest may also wane.
• A minority raise environmental/energy concerns, questioning whether the benefit justifies the resource usage.

Analogy and Meta‑Discussion

• Popular analogy: LLMs are to ML what Heroku or modern DevOps tools were to deployment—removing the need for big specialist teams but not fundamentally changing everything.
• Some foresee “preferred model wars” similar to past language/framework wars, as people adapt their own “dialect” of prompting to specific models.