GPT-4.5 or GPT-5 being tested on LMSYS?
A mysteriously labeled “gpt2-chatbot” model on the LMSYS Chatbot Arena is prompting speculation that OpenAI may be quietly testing a next-generation GPT-4–class system, possibly with a November 2023 training cutoff. Many users report it outperforming GPT‑4 and Claude Opus on coding, technical reasoning, niche knowledge, and translation, while others still find familiar weaknesses in math, factual reliability, and non‑English performance. The episode highlights both how hard it is to reliably benchmark LLMs in the wild and how quickly expectations are rising for incremental AI improvements.
Model access & basic behavior
- Model appears on LMSYS Chatbot Arena as “gpt2-chatbot”; direct-chat access is rate‑limited and often unavailable, so many test it via battle mode.
- When asked “what are you?”, it consistently replies that it is ChatGPT based on GPT‑4, with a knowledge cutoff around November 2023.
- Several commenters note this text is likely from a system prompt and not reliable “self‑knowledge.”
Perceived capabilities vs other LLMs
- Many report it outperforming GPT‑4‑turbo, Claude 3 Opus, Llama 3, Mixtral and others on:
- Coding (including obscure languages, hacklang, CodeMirror 6, in‑browser SVG→PNG, richer scaffolding and explanations).
- Technical domains (turbine blades, material science, DNA/SNPs, math-heavy engineering, hammered dulcimer pickups).
- Translation (including dialects), style imitation, and long, more “human‑like” prose.
- Practical reasoning (race fuel estimates with formation laps, health‑care policy design, assetto corsa fuel, etc.).
- Long‑tail web knowledge: old forums, niche car forums, obscure users, and some tiny personal sites.
Failures, errors & limitations
- Still makes clear mistakes:
- Arithmetic and modular arithmetic, Fermi estimates with many zeros, density/volume alcohol problems.
- Logic puzzles and word problems (Monty Hall variant, relatives/children puzzles, fruit ordering, river crossing).
- Some coding prompts (matrix “below secondary diagonal”, analog clock ticking precisely on real seconds, subtle Python name‑binding trick).
- Hallucinated scholarly citations and misdescribed studies; wrong or incomplete answers for niche technical tools and products.
- Several users say it “sounds” very confident and deep while missing parts of the task or inventing plausible but wrong details.
- Performance in non‑English languages is mixed: strong in some major languages/dialects, clearly worse in smaller ones.
Speculation about identity & training
- Hypotheses in the thread:
- A new OpenAI model (variously dubbed “GPT‑4.5”, “GPT‑5”, or a “GPT‑4 v2” with later cutoff).
- A GPT‑4‑class model with better data curation and more complete 2021–2023 coverage.
- A model with large‑scale retrieval or “memory” layered on top of GPT‑4.
- A non‑OpenAI model (e.g., large Llama 3 variant) fronted by LMSYS.
- No hard evidence emerges; participants repeatedly flag all attributions as speculative.
Why test via LMSYS?
- Some argue LMSYS provides the only large‑scale, head‑to‑head human preference benchmark in the wild.
- Others point out that A/B testing inside ChatGPT or with private testers would avoid leaks and brand‑bias, so using LMSYS is itself puzzling.
Meta: benchmarks, AGI, and progress
- Multiple ad‑hoc “personal benchmarks” are shared; results are highly task‑dependent and sometimes contradict each other.
- Broader debate arises about:
- How much LLMs have really improved over two years.
- Limits of next‑token prediction for precise reasoning and math.
- Definitions and “shades” of AGI, and whether current systems are just sophisticated language/search engines.