Gemini 4 Carbon is the new #1 on the Artificial Analysis Intelligence Index v5

Original Article ↗ Hacker News Discussion ↗

Google’s newly announced Gemini 4 Carbon model now tops the Artificial Analysis Intelligence Index, but many observers question the significance of a benchmark win for a system that isn’t yet publicly available. Commenters highlight a pattern of Google timing AI announcements around earnings, offering strong research and high-end models but pairing them with slow rollouts, tight usage limits, confusing product tiers, and frequent hallucinations in real use. Several also criticize benchmark aggregators like Artificial Analysis as increasingly opaque and commercially conflicted, arguing that leaderboard scores no longer align well with developers’ day‑to‑day experience, especially for coding workloads where Anthropic’s Opus models remain a favorite.

Perceived Quality of Gemini Models

  • Many say Gemini (especially 3.8 Flash and Argon) is strong on general knowledge, scientific explanations, math, and OCR/HTR, sometimes preferred over Claude or GPT for prose clarity and pedagogy.
  • Several find Argon comparable to or slightly below Opus 5.5 overall, and clearly behind top OpenAI/Anthropic models for coding.
  • Some see Gemini as a good “generalist” supervisor for multi-agent workflows, less so as a focused coding agent.

Access, Pricing, and Quotas

  • Repeated complaints that Google’s $20 “Pro” plan offers very low Argon usage (only a few medium requests before multi‑hour cooldown), making the frontier tier “practically unusable.”
  • Higher tiers ($99–$200, or bundled with large Google One plans) give more reasonable access, but users call them expensive.
  • Several say they default to other providers (Anthropic, OpenAI, Chinese labs) due to better limits and price/performance.

Carbon Announcement and Rollout Strategy

  • Strong criticism that Gemini 4 Carbon tops the Artificial Analysis index despite not being publicly available.
  • Many view the announcement as investor‑focused PR, timed to Google’s earnings, performance reviews, or safety theater.
  • Concern that by the time Carbon is accessible, competitors will have leapfrogged it and real‑world impact will be muted.

Real-world Use Cases and Comparisons

  • Positive: document analysis in industries like finance, trip planning with Maps integration, historical OCR, scientific/engineering reasoning, Flutter/Dart help.
  • Negative: weak tool/computer use (e.g., LaTeX/PDF workflows), poor integration with some Google surfaces (Calendar, Docs, My Maps).
  • Several report preferring Claude/Opus, GPT, DeepSeek, or Haiku for coding and agents, despite liking Gemini for explanations.

Hallucinations and Reliability

  • Many report unusually aggressive hallucinations from Gemini/Argon, even with access to Google Search: invented products, laws, paths in airports, map symbols, and fabricated transcripts.
  • Some say Gemini follows instructions and context poorly and degrades over a long session.

Views on Artificial Analysis and Benchmarks

  • Multiple commenters distrust AA: frequent index changes around big lab releases, removal or demotion of strong open‑weight/Chinese models, and heavy reliance on private benchmarks plus paid consulting.
  • Others argue evolving benchmarks and private test sets are necessary to combat “benchmaxxing,” but concede aggregate indices may no longer reflect day‑to‑day usefulness.