Qwen3.8-Flash-Next

Original Article ↗ Hacker News Discussion ↗

Qwen3.8-Flash-Next, a new mixture-of-experts large language model from Alibaba, is drawing attention for outperforming earlier Qwen releases while being much cheaper to train and serve, and for previewing the architecture planned for Qwen 4. It combines a 125B-parameter main model with a large n‑gram “engram” memory to separate factual recall from reasoning, promising strong quality at low active parameters and making it attractive for local deployment on high-RAM Macs, AMD Strix Halo systems, and DGX Spark boxes. Commenters are enthusiastic but note current limitations: tooling like llama.cpp and vLLM is only just adding support, quantized versions are large due to the n‑gram sidecar, and token overthinking and verbosity remain concerns compared to some frontier hosted models.

Model performance & positioning

  • Many see Qwen3.8-Flash-Next as outperforming Qwen3.8-27B and DeepSeek V4 Flash on benchmarks, especially given it’s a relatively small MoE in active parameters (≈6B per token).
  • Some are surprised it “beats” the dense 27B so cleanly; others note this is expected from a newer architecture and MoE design.
  • There is interest in how it compares to proprietary “frontier” models on cost-per-task, with some skepticism that token-based pricing alone tells the full story.

Architecture: MoE and N‑gram / “engram” embeddings

  • Architecture is described as 125B main params + ~51B N‑gram embeddings, with 6B active per token; seen as a preview of a future Qwen 4 family.
  • N‑gram/engram table is framed as:
    • A fact-recall sidecar that preserves specific knowledge without overloading dense weights.
    • Similar in spirit to RAG, but integrated into the network and not using context window.
    • A way to trade extra memory for less compute per token and more robust long-tail “world knowledge”.
  • Some clarify it’s not just “1‑bit quant” in practice; effective bit rates differ between core model and N‑gram table.

Local deployment & hardware

  • Strong excitement from users of Strix Halo, DGX Spark, high‑RAM Macs, and AMD/Apple platforms: 128 GB unified memory is a common target.
  • Reported speeds:
    • Strix Halo: ~20–35 tok/s now, with expectations of 50–60 tok/s with better kernels/MTP.
    • DGX Spark: ~12–20 tok/s decode, ~80–190 tok/s prefill with custom engines and NVMe‑paged N‑grams.
  • Concern: N‑gram sidecar (~50 GB+) makes it hard to run on 32–64 GB systems, undermining initial hope that a 6B‑active MoE would be CPU‑friendly.

Tooling, quantization & ecosystem

  • Early friction: mainline llama.cpp and vLLM initially lacked support; forks and special builds now exist, with PRs in progress.
  • Unsloth provides GGUF quants and documents that even “1‑bit” requires ~75+ GB RAM; higher‑bit quants can hit ~90+ GB.
  • Some complain that current quants overfocus on very low bits (1–2 bit), which may underrepresent model quality; others show decent results even at 2‑bit.

Reasoning style, overthinking & token efficiency

  • Multiple users report earlier Qwen 3.8 models “overthinking” at high reasoning levels, especially on open‑ended or one‑shot tasks, causing slow, verbose outputs.
  • Some hope Flash‑Next is more direct; early anecdotes are mixed: it can still second‑guess itself but often delivers strong results.
  • There is concern that Chinese “flash” models (including GLM and DeepSeek) can waste many tokens via long internal reasoning chains, which matters for API costs.

Pricing & economics

  • Qwen3.8-Flash-Next API pricing (e.g., $0.16 / $0.47 per million tokens) is compared against DeepSeek and OpenAI tiers.
  • Debate over whether inference at these prices is profitable:
    • One side argues inference is clearly margin‑positive and prices track broader market rates.
    • Another side suspects aggressive discounting, cross‑subsidies, or pre‑IPO positioning may mask true costs.

Knowledge vs. tools vs. architecture

  • Several comments digress into whether smaller models with web search or external knowledge stores can match larger models’ baked‑in “world knowledge”.
  • Some argue:
    • Large models encode a wide range of books/knowledge but don’t have perfect recall, so retrieval and citations still matter.
    • Tool‑augmented smaller models may be preferable when freshness and modular knowledge stores are important.
  • Others emphasize that you “can’t search what you don’t know exists,” so internal conceptual coverage still matters for discovery of algorithms and techniques.

Open-source dynamics & geopolitics

  • Thread notes contrast between relatively conservative US labs and aggressive, frequent open releases from Chinese teams (Qwen, DeepSeek, GLM).
  • Some see this rapid open release cycle as driving local‑LLM innovation and making new architectures (MoE + N‑grams, MTP, etc.) accessible quickly.
  • There’s an undercurrent that both tightly controlled frontier APIs and open‑weight Chinese models will coexist and push the field forward.