GPT-4.5 or GPT-5 being tested on LMSYS?

Original Article ↗ Hacker News Discussion ↗

Model access & basic behavior

  • Model appears on LMSYS Chatbot Arena as “gpt2-chatbot”; direct-chat access is rate‑limited and often unavailable, so many test it via battle mode.
  • When asked “what are you?”, it consistently replies that it is ChatGPT based on GPT‑4, with a knowledge cutoff around November 2023.
  • Several commenters note this text is likely from a system prompt and not reliable “self‑knowledge.”

Perceived capabilities vs other LLMs

  • Many report it outperforming GPT‑4‑turbo, Claude 3 Opus, Llama 3, Mixtral and others on:
    • Coding (including obscure languages, hacklang, CodeMirror 6, in‑browser SVG→PNG, richer scaffolding and explanations).
    • Technical domains (turbine blades, material science, DNA/SNPs, math-heavy engineering, hammered dulcimer pickups).
    • Translation (including dialects), style imitation, and long, more “human‑like” prose.
    • Practical reasoning (race fuel estimates with formation laps, health‑care policy design, assetto corsa fuel, etc.).
    • Long‑tail web knowledge: old forums, niche car forums, obscure users, and some tiny personal sites.

Failures, errors & limitations

  • Still makes clear mistakes:
    • Arithmetic and modular arithmetic, Fermi estimates with many zeros, density/volume alcohol problems.
    • Logic puzzles and word problems (Monty Hall variant, relatives/children puzzles, fruit ordering, river crossing).
    • Some coding prompts (matrix “below secondary diagonal”, analog clock ticking precisely on real seconds, subtle Python name‑binding trick).
    • Hallucinated scholarly citations and misdescribed studies; wrong or incomplete answers for niche technical tools and products.
  • Several users say it “sounds” very confident and deep while missing parts of the task or inventing plausible but wrong details.
  • Performance in non‑English languages is mixed: strong in some major languages/dialects, clearly worse in smaller ones.

Speculation about identity & training

  • Hypotheses in the thread:
    • A new OpenAI model (variously dubbed “GPT‑4.5”, “GPT‑5”, or a “GPT‑4 v2” with later cutoff).
    • A GPT‑4‑class model with better data curation and more complete 2021–2023 coverage.
    • A model with large‑scale retrieval or “memory” layered on top of GPT‑4.
    • A non‑OpenAI model (e.g., large Llama 3 variant) fronted by LMSYS.
  • No hard evidence emerges; participants repeatedly flag all attributions as speculative.

Why test via LMSYS?

  • Some argue LMSYS provides the only large‑scale, head‑to‑head human preference benchmark in the wild.
  • Others point out that A/B testing inside ChatGPT or with private testers would avoid leaks and brand‑bias, so using LMSYS is itself puzzling.

Meta: benchmarks, AGI, and progress

  • Multiple ad‑hoc “personal benchmarks” are shared; results are highly task‑dependent and sometimes contradict each other.
  • Broader debate arises about:
    • How much LLMs have really improved over two years.
    • Limits of next‑token prediction for precise reasoning and math.
    • Definitions and “shades” of AGI, and whether current systems are just sophisticated language/search engines.