Claude 3 beats GPT-4 on Aider's code editing benchmark

Original Article ↗ Hacker News Discussion ↗

Claude 3’s strong performance on a popular code-editing benchmark has prompted many developers to compare it directly with GPT‑4 in real-world programming tasks. Commenters highlight Opus’s advantages in long-context coding and reasoning, but note trade-offs such as higher cost, slower responses, stricter safety filters, regional availability limits, and occasional API instability. Overall, many see Claude 3 as a serious competitor that narrows or surpasses GPT‑4 in some areas, while others report inconsistent results and emphasize that model choice still depends heavily on use case, prompting style, and infrastructure.

Model quality and benchmarks

  • Claude 3 Opus slightly outperforms GPT‑4 on Aider’s code-editing benchmark and Chatbot Arena, but many see them as roughly on par in real use.
  • Some users find Opus clearly better for complex coding, long-context reasoning, and “stylish” writing; others report more hallucinations, API inventions, and wrong but confident answers than GPT‑4.
  • Several people run the same queries across Claude, GPT‑4, and Gemini and report: Claude ≈ GPT‑4, Gemini noticeably weaker.
  • Some think current models are near a “soft cap” for coding under current architectures, with further gains coming from tools like code execution, not raw model upgrades.

Coding use cases and workflows

  • Claude’s 200k-token context and better long-context recall are praised for whole‑codebase reasoning, refactors, and document analysis.
  • Others report poor performance on specific tasks (Terraform review, C#/Neovim questions, edge-case I/O behavior), sometimes worse than GPT‑4 or Mistral Large.
  • Many users see LLMs as excellent for boilerplate, unknown APIs, and small functions, but not reliable for larger rewrites, migrations, or end‑to‑end tests without heavy supervision.

Safety, alignment, and refusals

  • A major thread centers on Claude’s stronger guardrails: refusing to rephrase mildly charged language, write “beer song” code, or work in some repos.
  • Opinions split: some appreciate safety for misuse scenarios (propaganda, hacking, election interference); others see it as over‑cautious, paternalistic, and a practical deal‑breaker for production tools.
  • Users note substantial variance between Claude Instant/Sonnet/Opus on refusal behavior.

Cost, rate limits, and APIs

  • Claude 3 models are slower and more expensive than OpenAI’s, with stricter rate limits, especially in the chat UI; this pushes some toward OpenAI despite Opus’s quality.
  • Sonnet and Haiku are praised for price–performance; Haiku is cheap enough to enable “multiple‑samples plus voting” strategies.
  • Tools like OpenRouter, Aider, and various third‑party chat UIs are popular for swapping models and avoiding per‑vendor lock‑in.

Availability and regulation

  • Claude’s web UI is not officially available in the EU and some other regions, though API access is partly available and people report workarounds.
  • Many attribute staggered regional rollout to GDPR / EU AI regulation risk; this limits Claude adoption versus GPT, which is available in the EU.