Claude 3 beats GPT-4 on Aider's code editing benchmark

Original Article ↗ Hacker News Discussion ↗

Model quality and benchmarks

  • Claude 3 Opus slightly outperforms GPT‑4 on Aider’s code-editing benchmark and Chatbot Arena, but many see them as roughly on par in real use.
  • Some users find Opus clearly better for complex coding, long-context reasoning, and “stylish” writing; others report more hallucinations, API inventions, and wrong but confident answers than GPT‑4.
  • Several people run the same queries across Claude, GPT‑4, and Gemini and report: Claude ≈ GPT‑4, Gemini noticeably weaker.
  • Some think current models are near a “soft cap” for coding under current architectures, with further gains coming from tools like code execution, not raw model upgrades.

Coding use cases and workflows

  • Claude’s 200k-token context and better long-context recall are praised for whole‑codebase reasoning, refactors, and document analysis.
  • Others report poor performance on specific tasks (Terraform review, C#/Neovim questions, edge-case I/O behavior), sometimes worse than GPT‑4 or Mistral Large.
  • Many users see LLMs as excellent for boilerplate, unknown APIs, and small functions, but not reliable for larger rewrites, migrations, or end‑to‑end tests without heavy supervision.

Safety, alignment, and refusals

  • A major thread centers on Claude’s stronger guardrails: refusing to rephrase mildly charged language, write “beer song” code, or work in some repos.
  • Opinions split: some appreciate safety for misuse scenarios (propaganda, hacking, election interference); others see it as over‑cautious, paternalistic, and a practical deal‑breaker for production tools.
  • Users note substantial variance between Claude Instant/Sonnet/Opus on refusal behavior.

Cost, rate limits, and APIs

  • Claude 3 models are slower and more expensive than OpenAI’s, with stricter rate limits, especially in the chat UI; this pushes some toward OpenAI despite Opus’s quality.
  • Sonnet and Haiku are praised for price–performance; Haiku is cheap enough to enable “multiple‑samples plus voting” strategies.
  • Tools like OpenRouter, Aider, and various third‑party chat UIs are popular for swapping models and avoiding per‑vendor lock‑in.

Availability and regulation

  • Claude’s web UI is not officially available in the EU and some other regions, though API access is partly available and people report workarounds.
  • Many attribute staggered regional rollout to GDPR / EU AI regulation risk; this limits Claude adoption versus GPT, which is available in the EU.