GPT-4, without specialized training, beat a GPT-3.5 class model that cost $10M
A new paper claims that off‑the‑shelf GPT‑4 outperforms a $10M finance‑specific GPT‑3.5‑class model on most financial understanding benchmarks, reigniting debate over the value of domain fine‑tuning versus simply using larger general models. Commenters highlight that modern LLMs plus good prompting and retrieval (RAG) often beat expensive custom training, though narrow, task‑specific fine‑tunes can still excel in specialized domains like robotics or niche programming languages. The thread also touches on practical trade‑offs—cost, latency, context window limits, and tool support—as well as emerging rivals such as Claude and Mistral that sometimes match or exceed GPT‑4 on particular tasks.
GPT-4 vs $10M Finance Model
- Many see it as unsurprising that a much larger general model (GPT-4) outperforms a smaller, expensive, domain model on broad financial understanding.
- The result is framed as an instance of “scaling laws” and the “bitter lesson”: more compute and general training often beat handcrafted specialization.
- Some note caveats: GPT-4’s training data and fine-tuning are opaque, so claims of “no specialized financial training” are hard to verify.
- The finance benchmark largely measures reading comprehension in a specific domain; GPT-4 lags the finetuned model on narrower tasks like NER with specific labels.
Does Anything Beat GPT-4?
- Experiences are mixed: some users report Claude 3 Opus or Mistral-large beating GPT-4 on particular coding or programming tasks; others find GPT-4 more reliable.
- Chatbot Arena rankings are cited showing GPT-4 very slightly ahead of Claude 3 Opus, but differences are small.
- Users emphasize that performance varies by task, prompt style, and surrounding tools.
Tooling vs Model Capabilities
- Features like “write Python and run it” are attributed to tooling (system prompts + code execution environment), not innate model insight.
- Claude’s web UI currently lacks a Code Interpreter–style tool, leading some to misattribute weaknesses in math to the model rather than missing infrastructure.
Foundation Models, Fine-Tuning, and Scaling
- Debate over whether ChatGPT/GPT-4 is a “foundation model” or a heavily fine-tuned derivative, and how much that matters for interpreting results.
- Several argue that large, general models will usually beat smaller specialist models, but that finely tuned GPT‑4–class models could outperform in the future.
Prompting vs Fine-Tuning
- Strong claim: few-shot prompting with good examples on modern LLMs often beats fine-tuning, especially given low input-token costs.
- Others argue fine-tuning is essential for narrow tasks (e.g., niche programming languages, robotics) and can drastically outperform GPT‑4 in those domains.
- There is concern that naïve fine-tuning can “lobotomize” models or cause catastrophic forgetting, especially with smaller models.
RAG, Context Windows, and Private Data
- Some say RAG is becoming irrelevant with large context windows; others counter with concrete examples (millions of tokens of documents) where RAG is still necessary.
- Large context can reduce chunking complexity, but cost, latency, and irrelevance/noise remain issues.
- RAG is also seen as important for private or frequently updated data that can’t live in training sets.
Cost, Practicality, and Startup Strategy
- Opinions diverge on whether fine-tuning is “trivial”: mechanically easy with modern tools, but getting useful results is described as hard and error-prone.
- Claims appear that cheap LoRA/QLoRA fine-tunes can reach GPT‑4‑like performance for specific domains at far lower inference cost; other commenters demand concrete evidence.
- Several practitioners advise most companies not to train their own base models and to focus instead on prompt engineering, RAG, and integrating commodity LLMs into real products.
- Others argue that dismissing fine-tuning is shortsighted, particularly in robotics and high-specialization domains.