GLM-5.2 – How to Run Locally

Original Article ↗ Hacker News Discussion ↗

Enthusiasts are excited by the release of GLM‑5.2, an open‑weight large language model that approaches frontier systems like GPT‑5.x in quality, but quickly run into its enormous hardware demands: hundreds of gigabytes of RAM, high‑end GPUs, and heavy quantization to run “locally.” Much of the debate centers on whether it’s economically or practically sensible to self‑host such a model—balancing degraded performance from low‑bit quantization, slow token speeds, and high power costs against privacy, control, and long‑term savings versus API fees. Many expect that in the near term the sweet spot will remain smaller 20–80B models, while future hardware (unified memory Macs, AI workstations, denser GPUs) and more efficient architectures could eventually make GLM‑class models viable outside datacenters.

Hardware Requirements & Feasibility

  • GLM‑5.2’s full model needs on the order of 750 GB–1.5 TB (FP8–FP16); “local” here usually means high‑end workstations or servers, not typical desktops or laptops.
  • Realistic setups discussed: 512 GB DDR4/DDR5 + dual RTX 3090/RTX 6000 or similar; Mac Studio with 256–512 GB unified RAM; DGX Spark–class boxes; upcoming high‑RAM APUs (Strix/Medusa Halo) or Intel/Crescent Island–style accelerators.
  • Many argue 256 GB RAM is a theoretical minimum; 512 GB+ is “realistic” for usable speeds and less extreme quantization.

Performance, TPS, and Memory Bandwidth

  • Decode speed is repeatedly framed as “memory‑bandwidth math”:
    TPS ≈ active weights (GB) ÷ memory bandwidth (GB/s).
  • With 40B active parameters at 4‑bit (20 GB) and ~100 GB/s bandwidth, ~5 tok/s is expected; MoE routing and speculation (MTP) can improve or hurt depending on bottlenecks.
  • Reported local speeds:
    • GLM‑5.2 Q4 on mixed CPU+2×3090: ~6 tok/s (can climb with faster RAM/CPUs).
    • CPU‑only Q6: ~1 tok/s; 16 parallel streams don’t raise per‑stream TPS.
  • Heavy offloading to CPU or disk is described as “unusable” for interactive work, especially due to very slow prompt prefill.

Quantization vs Quality

  • Thread emphasizes that benchmark claims like “4‑bit dynamic is essentially lossless” can be misleading; KL divergence ≠ real‑task quality.
  • Several users say they must go 1–2 bits higher (Q5/Q6) than claimed “lossless” Q4 for long‑context, complex tasks.
  • GLM‑5.2’s advertised near‑GPT‑5.x performance only applies at FP8/FP16; at FP4/Q4 loss is modest, but at very low bitwidth (FP2/Q1–Q2) quality reportedly degrades to below strong mid‑tier frontier models.

Local vs Cloud Economics

  • For individuals, running GLM‑5.2 locally at high quality is often judged “not economic”: hardware $10k–$90k, plus hundreds of watts of power, versus relatively cheap API access.
  • For teams or companies already spending thousands/month on tokens, a $15k–$80k on‑prem server can break even in 1–3 years, especially with shared use and predictable privacy/compliance.
  • Electricity cost comparisons show local inference can be cost‑comparable per token, but only after substantial capex.

Privacy, Control, and Motivation

  • Strong interest in local models for:
    • Data privacy and avoiding vendor logging/denials of service.
    • Regulatory and geopolitical insulation.
    • Freedom from rate limits, caps, and “enshittification” of cloud products.

Competition, Model Landscape & Future

  • Many see GLM‑5.2 as part of a broader open‑weights wave (alongside DeepSeek, Qwen, etc.) eroding claims of proprietary “moats.”
  • Consensus: near‑term “sweet spot” for consumers is still ~27–35B models (e.g., Qwen3.6‑27B) fully in VRAM; 750B‑class models remain niche, expensive, and often too slow locally.