Run Qwen 3.8 Flash Next (125B) on consumer hardware (RTX 4090) at 100T/s

Original Article ↗ Hacker News Discussion ↗

Running Qwen 3.8 Flash Next (a 125B-parameter MoE model) on consumer GPUs with the Strata inference engine is giving users 50–120+ tokens per second, leading some to say it rivals or replaces cloud models like Claude Opus for day-to-day coding and agentic work. Much of the debate centers on aggressive low‑bit quantization (Q2–Q3): enthusiasts report surprisingly little quality loss versus Qwen 3.8 27B and even Opus‑4.7‑level performance, while skeptics highlight degradation in harder tasks, vision accuracy, and long‑horizon reliability. Overall, the project is seen as a major step in making high‑end local LLMs practical on ~$1.5–3k hardware, even as questions remain about accuracy, concurrency, and how far sub‑4‑bit quantization can be pushed.

Performance and Hardware Results

  • Multiple users report 100–125 tok/s on RTX 4090 with Qwen 3.8 Flash Next using Strata, even with large contexts (up to 262k+).
  • Older/less powerful setups still see usable speeds: ~30 tok/s on 3080 10GB, ~40–60 tok/s on 3090, ~60 tok/s on an R9700 with large system RAM, ~7 tok/s CPU-only but useful for long overnight runs.
  • Apple silicon (M1/M5 Max, 64–128GB) runs 3-bit quants at 20–70 tok/s, but CPU bandwidth and RAM limits can bottleneck prefill.
  • Some report 4-bit quants of Flash Next via other engines (e.g., ds4) reaching 200+ tok/s decode on high-end workstation GPUs.

Quantization and Model Variants

  • Thread heavily debates low-bit quants:
    • Q2: widely viewed as too degraded for serious work, but some claim larger models and good calibration can make Q2 usable for narrow tasks.
    • IQ3_XXS / IQ3_S: repeatedly cited as a “sweet spot,” with one benchmark claiming within ~1 point of unquantized, sometimes even slightly better on some tasks.
    • Q4: many consider this the practical floor if you want to “retain most function.”
  • “Coder” variant prunes half the experts and hits ~91% of full SWE-Bench Verified but several users feel it is noticeably worse than the full Flash Next model.

Quality vs 27B and Cloud Models

  • Many report Flash Next (even at 3-bit) clearly outperforming Qwen 3.8 27B for coding and general tasks, while also being faster per token.
  • Others say 27B at higher precision (e.g., Q6, FP8/NVFP4) feels more accurate or more stable, especially at very long contexts.
  • Some users benchmark Flash Next (IQ3 quants) as roughly on par with prior frontier cloud models (various Opus/Sonnet versions) on coding benchmarks like DeepSWE; others remain unconvinced and prefer cloud for reliability.

Inference Engines and Implementation Details

  • Strata’s MoE expert caching/offload is the key novelty: it keeps ~6B active parameters per token while paging experts between VRAM and RAM.
  • Users compare Strata to llama.cpp, ds4, ninfer, sglang, and FreeToken; Strata often wins on speed for Flash Next but can be worse on some tasks (notably one vision benchmark where llama.cpp significantly outperformed Strata with the same weights).
  • Concurrency: initial reports of poor parallel behavior are corrected; Strata supports a “parallel” setting and now swaps contexts to RAM.

Limitations, Skepticism, and Open Questions

  • Several commenters distrust sub‑4‑bit quants and point out that many “100 tok/s on consumer GPU” demos omit accuracy metrics.
  • Vision quality under Strata is specifically reported as much worse than llama.cpp on a coordinate‑prediction test.
  • Some users see hallucinations or tool‑calling issues at low bitrates or with aggressive MTP; others report no such problems and stress configuration sensitivity.
  • There is debate over whether Flash Next Q2/Q3 is truly better than a well‑quantized dense 27B, especially on constrained hardware.

Broader Themes

  • Strong enthusiasm that “Opus‑like” local performance is now reachable on ~$1.5–3k consumer setups, though not yet for everyone.
  • Concerns about hardware cost, VRAM/RAM stagnation, and quantization hype; some expect further efficiency gains to undercut the current datacenter build‑out.
  • Discussion touches on security (curl|bash installers, local agents) and censorship/coverage of politically sensitive topics, with examples that Qwen models can still recall some sensitive historical events when uncensored.