Unsloth Dynamic 3.0 GGUFs
Unsloth’s new “Dynamic 3.0” GGUF quantizations for Qwen3.8-27B aim to shrink model sizes dramatically while preserving most of their accuracy, enabling large LLMs to run on 16–32GB consumer hardware. Commenters explore trade‑offs between extreme low‑bit quants (down to 1–2 bits), speed, context length, and coding quality, with some reporting useful results and others seeing looping or degraded outputs, especially at Q2 levels. There is strong interest in better benchmarking of real tasks, improved multi‑GPU and Apple MLX support, and clearer versioning, as users weigh these quants against alternatives like ExLlama and cloud‑hosted full‑precision models.
Dynamic 3.0 changes & MTP trade-offs
- Very small GGUFs (<~8 GB, e.g., IQ2_XXS and below) ship without MTP to save ~500–750 MB, which matters on 8–16 GB systems.
- Some users question removing a speed feature from models aimed at low-resource devices; others argue that at Q2 the model is so degraded that extra memory is better spent on a higher-bit quant than on MTP.
- A separate Q4_0 MTP drafter is available for those who want it, but the recommendation for 16 GB machines is to use higher-bit small quants (e.g., IQ3_XXS, Q2_K_XL) rather than extreme Q2.
Quality of low‑bit quants
- Reported experience with 1–2 bit and bonsai quants ranges from “surprisingly usable for light coding” to “essentially useless” on stricter private evals, where small errors cascade.
- Some suggest dropping to a smaller 9B-class model instead of pushing a 27B to 1–2 bits.
- New UD 1‑bit quants claim ~72% top‑1 accuracy at ~89% size reduction, but real‑world validation is still sparse.
Performance on local hardware
- 16 GB GPUs: users debate best tradeoff between Unsloth UD3.0 GGUF vs ExLlama 4-bit; ExLlama can keep embeddings in system RAM and may beat earlier UD2.0 Q4 in quality.
- Multi‑GPU: tensor and layer splits across 2–8 GPUs are used; performance depends heavily on PCIe lanes and power constraints.
- Macs (M1–M4): 27B Qwen runs but can be slow (20–40 tok/s). Some find Ollama’s MLX builds with MTP faster than llama.cpp setups.
Benchmarks, KL divergence & “doom loops”
- Some criticize KL divergence or similar metrics as not capturing long-horizon behavior and compounding errors.
- Others argue errors don’t simply multiply, since models can self‑correct in chain‑of‑thought.
- Unsloth introduced “Divergence‑300” (32/512-token inference tests on held‑out tasks) and plans broader benchmarks, but time/compute budgets seem tight.
- Users report fewer “doom loops” and more self‑correction in Qwen3.8‑27B vs 3.6, though it can “think in circles” and be verbose.
Usability, formats, and tooling
- Confusion arises because Dynamic 3.0 reuses existing file names; checksums show some advertised “new” files are unchanged. Calls for explicit versioning and richer GGUF metadata.
- Some prefer managing models via Hugging Face CLI or git‑LFS for history; others dislike extra tooling and stick to curl + checksums.
- Quantization itself is reported as fast and feasible on consumer hardware (including Macs), via llama.cpp tools or similar compressors.