Fine tune a 70B language model at home

Original Article ↗ Hacker News Discussion ↗

Scope: “Training” vs Fine‑tuning

  • The system lets users fine‑tune existing 70B models on 2×24GB GPUs, not pretrain from scratch.
  • Fine‑tuning updates a small fraction (~2%) of parameters, leveraging the base model’s language and world knowledge.
  • Some argue the conceptual gap between pretraining and fine‑tuning is overdrawn; “continued pretraining” is a better framing.
  • Training a 70B model from scratch still requires ~80,000 GPU‑hours and significant electricity cost; out of reach for most home users.

Technical Methods & Open Questions

  • QLoRA (4‑bit quantization + LoRA adapters) plus FSDP enables multi‑GPU, memory‑efficient fine‑tuning.
  • Recent <4‑bit methods (incl. ternary / BitNet‑style (-1,0,1)) are discussed as promising, especially for inference; adapting them to fine‑tuning seems feasible in principle but requires new kernels.
  • Sparsification (especially block sparsity) and hybrid sparse‑quantized schemes (e.g., SpQR) are seen as complementary but complex in practice.
  • Questions remain about QLoRA downsides: quantization degrades the base model before fine‑tuning; effects outside the finetuning domain are unclear.

Hardware, Platforms, and Distributed Training

  • 2×24GB GPUs are still considered expensive but far cheaper than 80GB datacenter GPUs; work is ongoing to reach dual‑16GB setups.
  • FSDP is preferred over DeepSpeed for hackability, though DeepSpeed is seen as feature‑rich and solid elsewhere.
  • Multi‑node training over networks is supported (e.g., via Slurm). Ideas for “LLM@home”‑style distributed community training are raised but bandwidth and software architecture are concerns.
  • Apple Silicon has attractive memory bandwidth and unified memory for large inference, but training support and raw compute lag Nvidia. AMD looks cost‑effective but tooling and drivers lag.

Use Cases: RAG vs Fine‑tuning

  • Many consider RAG better for injecting new factual knowledge; fine‑tuning excels at style, format, behavior, and small domain shifts.
  • There’s interest in domain‑specific assistants (e.g., literature, fiction writing, personal data, U‑Net/diffusion models) and “AI OS” scenarios; the optimal mix of RAG vs small‑scale fine‑tuning is viewed as unresolved.

Risk, Energy, and Open‑Source Ecosystem

  • Strong enthusiasm for democratizing powerful models and reducing concentration of AI power.
  • Debate over societal risks: some see this as no more dangerous than releasing new tools; others argue new capabilities imply nontrivial risk and warrant more reflection.
  • Energy use draws criticism; replies note efficiency gains, comparison to other high‑energy activities (gaming, flying, crypto), and emphasize clean‑energy scaling over usage policing.
  • Contributors encourage non‑experts to help via documentation, tooling, datasets, and community support to keep state‑of‑the‑art AI open.