Benchmarking GPT-4 Turbo – A Cautionary Tale
Benchmarks comparing OpenAI’s new GPT‑4 Turbo model with the original GPT‑4 suggest Turbo may perform slightly worse on certain coding exercises, likely because it “remembers” fewer training examples while offering faster, cheaper inference and much longer context. Commenters debate whether success on standardized problem sets reflects genuine reasoning or mere memorization, and how to design fair tests when models are trained on vast swaths of internet data. Others share real‑world results ranging from slight quality drops to noticeable improvements, and note broader issues like statistical rigor in evaluations, shifting developer workflows, and the ethics of reusing open source benchmarking setups.
Overall theme
- Discussion centers on what the Exercism benchmark actually measures in GPT‑4 vs GPT‑4 Turbo, and whether differences reflect memorization, reasoning ability, or noise.
- Many see Turbo as “faster but a bit dumber”; others argue the evidence is weak or that Turbo is actually better in their use cases.
Memorization vs reasoning
- Some interpret the results as: GPT‑4 has more benchmark problems memorized; Turbo “forgets” more but reasons similarly on unseen tasks.
- Others argue that identifying problems from just title + function stub can often be done via general knowledge, not exact memorization.
- Several commenters stress that memorization is a core part of intelligence (human or machine) and not inherently “cheating.”
- Concern is raised that when LLMs solve benchmarks they might mostly be recalling training data rather than demonstrating broad capability.
Can LLMs really code?
- Strong disagreement: some claim success on known problems is just “cheating,” hence doesn’t prove coding ability.
- Others provide anecdotes of LLMs successfully writing tests and implementations for novel, private codebases and domains.
- Several note that a large fraction of human programming is also pattern recall; LLM assistance is therefore still valuable.
Benchmark design & statistical strength
- Multiple comments note the sample size (~67 questions) is too small; with binomial confidence intervals, the GPT‑4 vs Turbo gap is statistically weak.
- People call for larger, more realistic benchmarks (e.g., multi-file tasks, long-context coding) and for multiple runs to handle randomness.
Reported user experiences
- Some users see GPT‑4 outperform Turbo on complex SQL, ML-paper analysis, and custom visual Q&A datasets.
- Others report Turbo is slightly more accurate and significantly faster on tasks like OCR-to-structured-text conversion or Exercism-style coding via other tools.
- Aider’s separate Exercism benchmark (Python) reportedly shows Turbo doing better than earlier GPT‑4 variants, contradicting the original blog’s pattern.
Context windows, costs, and behavior
- Turbo’s long context is praised, but several note degraded reliability beyond ~30–40k tokens.
- Questions arise about why completion length is capped at 4,096 tokens despite large context; responses suggest cost and latency constraints.
- Some speculate Turbo may be a smaller/distilled model, trading some quality for speed and cheaper inference.
Ethics, licensing, and evaluation concerns
- Worries about test contamination: closed models can’t easily be checked for training-set overlap and might overfit public benchmarks.
- A side thread discusses open-source attribution and code reuse between benchmarking tools.
- Some criticize model “nerfing” after users have paid, and compare LLM benchmark gaming to hardware benchmark optimization.