Qwen3.8 27B scores 52 on Artificial Analysis

Original Article ↗ Hacker News Discussion ↗

A new 27‑billion‑parameter open model, Qwen3.8 27B, is scoring on par with far larger frontier systems such as GLM 5.2 and GPT‑5.6 Luna on the Artificial Analysis benchmark, while remaining small enough to run on high‑end consumer GPUs. Commenters highlight its strong agentic behavior, low hallucination rate, and impressive coding performance, albeit at the cost of heavy “overthinking” and high token usage that impact speed and hosting costs. The thread reflects broader trends: increasingly capable local models are closing the gap with proprietary giants, raising questions about benchmark validity, infrastructure economics, and whether massive data‑center‑scale models will remain worth their escalating cost.

Model performance & benchmarks

  • Qwen3.8 27B scores 52 on Artificial Analysis, on par with much larger models like GLM 5.2, GPT‑5.6 Luna, and DeepSeek V4 Flash 0731.
  • It beats all “medium” open models (40B–150B) and previous Qwen3.6 27B (score 38), and is described as “frontier-adjacent” in code/agent tasks.
  • Some argue it appears “benchmaxxed” due to large variance across sub‑benchmarks; others point to broader benchmark aggregators where it is more consistently strong.
  • Several posters say head‑to‑head human ELO rankings (e.g., arena-style) may be more representative than synthetic benchmarks.

Reasoning behavior & token usage

  • The model uses substantially more tokens per task than many peers (≈2x relative to some models), especially at “max” reasoning levels.
  • Users describe it as highly agentic, persistent, even “obsessive,” sometimes overthinking or taking very long traces; some report fewer looping issues than Qwen3.6, others still see long, indecisive runs.
  • Non‑hallucination rate is highlighted as unusually high (~70%), making it attractive for tool‑calling and web‑augmented workflows.
  • There is debate on whether this is “overthinking” versus necessary test‑time scaling for a smaller dense model.

Local deployment & hardware

  • A major theme is excitement that a 27B dense model with near‑frontier capability can run on consumer GPUs (single high‑end card or dual 3090/4090 setups, 4‑bit/8‑bit quant).
  • Some benchmark it as slower in tokens/sec than comparable Gemma models; others show similar speeds under specific runtimes and settings.
  • Users note significant behavioral differences between Q4 and Q8 quants: Q4 makes more mistakes and compensates with extra “thinking.”

Pricing, architecture & hosting

  • Despite its smaller size, hosting prices on some APIs are higher than for much larger MoE models.
  • Explanations given:
    • Dense 27B means more active parameters and higher compute per token.
    • Qwen’s KV cache is less memory‑efficient than DeepSeek V4 Flash, making high‑concurrency serving costlier.
    • Official vendor pricing and profit margins also influence API rates.

Real‑world use & comparisons

  • Several users report strong performance on custom coding/agent workflows, sometimes rivaling or exceeding larger proprietary models, but with slower end‑to‑end latency.
  • Others find Chinese open models still weaker on unusual, cross‑domain creative tasks compared to top proprietary models.

Broader implications

  • Many see this as a major win for open‑weight, locally runnable models and question the long‑term value of huge proprietary data centers if “good enough” fits on consumer hardware.
  • Some predict regulatory pressure against strong open models if they continue to undercut heavily financed US labs.