Building a deep learning rig
Building a personal deep learning rig is framed as a trade-off between upfront hardware cost, cloud rental fees, and the practical benefits of having local compute. Commenters compare consumer GPUs, mining rigs, and boutique systems like Tinybox, debating Nvidia’s mature CUDA ecosystem versus AMD’s cheaper but more fragile ROCm stack. Many argue that despite sometimes weaker pure economics, owning local GPUs dramatically lowers friction for experimentation, data privacy, and long-running AI workloads.
Tinybox and AMD vs. Nvidia
- Some are eyeing Tinybox (AMD-based) as an alternative to Nvidia, but others warn it means “wasting time” on ROCm vs. getting work done with CUDA.
- Strong criticism that ROCm is fragile, slow vs. specs, and costs weeks of setup; others counter that recent releases (e.g., 7900XTX support) are “not that bad” and improving.
- Debate over whether Nvidia’s dominance is primarily from superior long-term software investment vs. anticompetitive behavior and ecosystem lock-in.
- Tinybox is seen by some as just off‑the‑shelf parts with markup; others argue the integration, debugging (e.g., PCIe AER errors), and system design are non‑trivial value adds.
GPU Hardware Choices
- 4090 vs Ada workstation cards: workstation variants run lower power (≈300W vs. 450W), can be dual‑slot, and support P2P, making 4‑GPU “normal case” builds easier.
- Many note 4090s can be power-limited to ~300W with little performance loss.
- For >2 GPUs, workstation/EPYC/Threadripper platforms and mining-style open frames or cases are common themes.
- PCIe x8 is generally seen as sufficient for most training/inference; x1 is too slow. NVLink often considered overkill for small multi‑GPU home rigs.
Local Rig vs. Cloud Economics
- Example numbers: 3×3090 on vast.ai at ~$0.6/hr vs. buying used 3090s; breakeven around ~128 days of full utilization, adjusted downward by resale value and GPU price stagnation.
- Electricity costs and non‑depreciating GPUs strengthen the case for ownership if utilization is moderate to high.
- Counterpoint: many hobbyists won’t hit breakeven vs. cheap community clouds; cloud can be cheaper if local power is expensive.
Ergonomics and Experimentation
- Several emphasize the “mental friction” of hourly-billed cloud: people run fewer weird or exploratory experiments there.
- Local rigs feel “free” once bought; users are more likely to queue overnight runs and tinker frequently.
Cloud Providers & Practicalities
- vast.ai and similar platforms are praised for price but criticized for reliability, variable bandwidth, potential “scams,” and data movement pain.
- runpod is mentioned as a somewhat more reliable alternative.
Other Hardware & Setups
- Threadripper/EPYC/ARM (Ampere Altra) suggested for maximizing PCIe lanes, though ARM brings tooling/compile friction.
- Mac M‑series can do inference but are far slower and less cost‑effective than desktop GPUs, especially for training.
- Multiple posts share or seek guidance on motherboards, mining frames, dual‑4090 builds, and small home rigs used for both gaming and local LLM/diffusion.