GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M

Original Article ↗ Hacker News Discussion ↗

GPT-4 vs $10M Finance Model

  • Many see it as unsurprising that a much larger general model (GPT-4) outperforms a smaller, expensive, domain model on broad financial understanding.
  • The result is framed as an instance of “scaling laws” and the “bitter lesson”: more compute and general training often beat handcrafted specialization.
  • Some note caveats: GPT-4’s training data and fine-tuning are opaque, so claims of “no specialized financial training” are hard to verify.
  • The finance benchmark largely measures reading comprehension in a specific domain; GPT-4 lags the finetuned model on narrower tasks like NER with specific labels.

Does Anything Beat GPT-4?

  • Experiences are mixed: some users report Claude 3 Opus or Mistral-large beating GPT-4 on particular coding or programming tasks; others find GPT-4 more reliable.
  • Chatbot Arena rankings are cited showing GPT-4 very slightly ahead of Claude 3 Opus, but differences are small.
  • Users emphasize that performance varies by task, prompt style, and surrounding tools.

Tooling vs Model Capabilities

  • Features like “write Python and run it” are attributed to tooling (system prompts + code execution environment), not innate model insight.
  • Claude’s web UI currently lacks a Code Interpreter–style tool, leading some to misattribute weaknesses in math to the model rather than missing infrastructure.

Foundation Models, Fine-Tuning, and Scaling

  • Debate over whether ChatGPT/GPT-4 is a “foundation model” or a heavily fine-tuned derivative, and how much that matters for interpreting results.
  • Several argue that large, general models will usually beat smaller specialist models, but that finely tuned GPT‑4–class models could outperform in the future.

Prompting vs Fine-Tuning

  • Strong claim: few-shot prompting with good examples on modern LLMs often beats fine-tuning, especially given low input-token costs.
  • Others argue fine-tuning is essential for narrow tasks (e.g., niche programming languages, robotics) and can drastically outperform GPT‑4 in those domains.
  • There is concern that naïve fine-tuning can “lobotomize” models or cause catastrophic forgetting, especially with smaller models.

RAG, Context Windows, and Private Data

  • Some say RAG is becoming irrelevant with large context windows; others counter with concrete examples (millions of tokens of documents) where RAG is still necessary.
  • Large context can reduce chunking complexity, but cost, latency, and irrelevance/noise remain issues.
  • RAG is also seen as important for private or frequently updated data that can’t live in training sets.

Cost, Practicality, and Startup Strategy

  • Opinions diverge on whether fine-tuning is “trivial”: mechanically easy with modern tools, but getting useful results is described as hard and error-prone.
  • Claims appear that cheap LoRA/QLoRA fine-tunes can reach GPT‑4‑like performance for specific domains at far lower inference cost; other commenters demand concrete evidence.
  • Several practitioners advise most companies not to train their own base models and to focus instead on prompt engineering, RAG, and integrating commodity LLMs into real products.
  • Others argue that dismissing fine-tuning is shortsighted, particularly in robotics and high-specialization domains.