Qwen2.5-1M: Deploy your own Qwen with context length up to 1M tokens

Original Article ↗ Hacker News Discussion ↗

A new 7B-parameter Qwen2.5 model with a 1 million token context window is prompting excitement among people trying to run full codebases and large documents locally, especially on Apple Silicon Macs via tools like Ollama and MLX. Many report that the real bottlenecks are memory usage, compute time, and tooling defaults: long contexts can require over 100GB of VRAM or unified memory, and frameworks may silently truncate input or become unstable without careful tuning and quantization. Others note that even when long contexts work technically, model quality often degrades beyond ~30–100K tokens, suggesting that smarter retrieval, agentic workflows, or new architectures may matter more than simply pushing context limits higher.

Running Qwen2.5-1M Locally (Mac, MLX, GGUF, CPU)

  • People are experimenting with running very long prompts (hundreds of thousands of tokens) on Macs, especially M3/M4 Max with 64–128GB unified memory.
  • One report: ~446k-token Rust/TypeScript codebase query on an M4 Max ran ~4 hours and returned a seemingly reasonable answer.
  • MLX 4-bit variants exist for macOS, but current MLX doesn’t yet support the dual-chunk attention mechanism used for full 1M-token context.
  • Some consider trying 1M-token prompts on large-RAM CPU servers, but expect it to be extremely slow.

Memory, Context Length, and KV Cache

  • Long context is dominated by KV cache memory, which scales with sequence length; 1M tokens requires “obscene” amounts of RAM/VRAM.
  • Official guidance:
    • Qwen2.5-7B-Instruct-1M: ~120GB VRAM for full 1M context.
    • Qwen2.5-14B-Instruct-1M: ~320GB VRAM.
  • KV cache quantization (e.g., 4-bit) can cut cache memory to ~¼, at the cost of quality.
  • Comparison table for another model shows memory rising steeply as context grows (e.g., ~27.5GB→109.8GB going from 200k @4-bit→16-bit).

Ollama Defaults and Context Configuration

  • Ollama’s num_ctx defaults to 2k and is widely seen as a “foot-gun”: it silently discards leading tokens when exceeded.
  • Users must explicitly set num_ctx higher or save a model variant with increased context. Documentation and behavior are criticized as confusing.
  • Plugins and tools (e.g., files-to-prompt integrations) support passing num_ctx, but users often misinterpret it as output length.

Hardware Choices and Cost/Accessibility

  • Macs with large unified memory are attractive for big-context local inference, but RAM configurations are expensive and tightly coupled to CPU tiers.
  • Some argue multi-GPU x86 builds (e.g., multiple 3090s) offer better raw compute per dollar, but they lack the large unified memory of Apple Silicon.
  • Debate over class/access: high-RAM Macs and large GPU rigs are seen as increasingly out of reach for many, shifting experimentation back toward the well-funded.

Actual Usefulness of Huge Context Windows

  • Multiple reports say models often degrade beyond ~25–32k tokens for coding and other precise tasks: loss of instruction-following, missed files already in context, poor recall.
  • Others counter that 1M–2M context in some services works well for high-level overviews or summarization of large codebases or corpora.
  • Overall sentiment: large context is promising but unreliable for complex, fine-grained tasks; retrieval quality and “lost in the middle” remain major issues.

Benchmarks, Long-Context Hype, and Limits

  • Skepticism about “nearly perfect” long-context claims: detailed tables show significantly less than 100% on complex tasks and often only up to 128k, not full 1M.
  • Long context is distinguished from generation length; several commenters note that output length across turns is still a hard, unresolved problem.