How to run Qwen 3.5 locally

Original Article ↗ Hacker News Discussion ↗

Advances in the open Qwen 3.5 models are making locally run LLMs genuinely useful for coding, OCR, and personal assistants on consumer hardware ranging from RTX 30/40-series GPUs to Apple Silicon and even low-power devices. Commenters trade notes on which parameter sizes and quantization levels (e.g., 9B vs 27B vs 35B-A3B, Q4 vs Q6) give the best balance of quality, speed, context length, and VRAM usage, and how to integrate them with tools like llama.cpp, LM Studio, and Claude Code. While many find Qwen 3.5 impressive and cost-saving compared to hosted APIs for many tasks, it still lags top frontier models for complex, long-horizon, or highly reliable agentic work, and setup remains too technical for non-experts.

Capabilities and Use Cases

  • Many users find Qwen3.5 surprisingly strong for local coding: multi-file edits, building small Rust apps, HTML/CSS work, and “rubber duck” explanations of code and compile errors.
  • 9B and 4B/0.8B variants are used for OCR, text cleanup, simple coding, translation, and log triage, but are considered weak for complex coding or “agentic” multi-step tasks.
  • Qwen models are also used for structured text/image/video analysis (JSON output, NER, categorization) and financial data extraction from emails.
  • Several note that Qwen3.5 is very confident and often sycophantic; personas (e.g., “brief rude senior”, “emotionless vulcan”) help tone it down.

Performance, Hardware, and Context

  • Reports span from tiny SBCs and phones (0.8B/2B CPU-only) to RTX 30/40/50-series, Apple M1–M5, and A100/H100.
  • 9B often reaches ~60–100 tok/s on mid/high-end GPUs; 35B-A3B can run at ~14–25 tok/s on consumer cards with partial offload.
  • 27B and 35B-A3B are widely seen as the sweet spot for “serious” local coding, with 27B sometimes stronger on benchmarks, 35B-A3B faster due to MoE.
  • Long contexts (100k–256k) are possible but some see quality degradation and instruction drift over long sessions, partly due to sliding-window attention.

Quantization and Model Choice

  • Frequent discussion of Q4_K_M / UD-Q4_K_XL vs other 4-bit schemes; Q4_0 and Q4_1 are described as faster but notably less accurate.
  • Rule of thumb in the thread: more parameters at lower bit-depth usually beats fewer parameters at higher bit-depth, down to ~3–4 bits.
  • Users recommend experimenting: 27B at 4-bit or 35B-A3B at 3-bit for 16GB+ VRAM; 9B or 4B for very constrained setups.

Thinking Mode and Reliability

  • “Thinking” mode can run indefinitely or add large latency; multiple users disable or heavily constrain it.
  • Some note improved reasoning and code review with thinking enabled; others find loops, repetition, or crashes in certain orchestrators.

Comparison to Frontier and Overall Sentiment

  • Enthusiasts claim 27B/35B approximate older Claude/GPT tiers or at least Haiku-level for many coding tasks, and can save substantial API costs.
  • Skeptics report that even 122B/397B lag behind top proprietary models (e.g., Claude Opus/Sonnet) on hard coding and mathematical/physical reasoning, and still hallucinate.
  • Consensus: Qwen3.5 is a major step for local models, excellent for many targeted workflows, but not yet a full replacement for state-of-the-art hosted models—especially for complex, long-horizon “agentic” coding.