So you want to rent an NVIDIA H100 cluster? 2024 Consumer Guide

Original Article ↗ Hacker News Discussion ↗

High‑end AI GPU clusters such as NVIDIA H100 and AMD MI300X are increasingly rented rather than bought, but buyers face opaque, case‑by‑case pricing and complex trade‑offs around networking, reliability, and support. Commenters compare InfiniBand and Ethernet fabrics, argue over used vs. fully supported hardware, and note that large cloud providers often can’t guarantee affordable, tightly coupled H100 capacity for serious training runs. Environmental impact and power sourcing also factor into provider choice, with some operators emphasizing “green” data centers and others skeptical that individual purchasing decisions meaningfully affect global emissions.

Pricing, Transparency, and Economics

  • Several commenters note the article is light on concrete prices; many vendors insist on private, quote-based deals.
  • One data point: median H100 Infiniband cluster price from a meta-source is around $2.3–2.47 per GPU-hour.
  • A back-of-envelope for 256 H100s at $2.47/h is ~$455k/month, implying ~21 months to recoup $10M of GPUs (excluding power, infra, and ops) — seen as a tough business with rapid depreciation risk.
  • Discussion highlights deliberate price discrimination in high-end markets and the difficulty of “price discovery,” especially for non-technical buyers.

Networking: InfiniBand vs Ethernet

  • InfiniBand is reported to systematically outperform Ethernet for large distributed training (3–10% at 16 nodes; gap widens with scale) and to be more reliable.
  • Ethernet-based RoCE clusters can work, but require careful design (lossless Ethernet, separate compute vs storage networks, avoiding packet reordering).
  • Lead times for current-gen IB switches are reported as ~50+ weeks, pushing some to Ethernet.
  • PCIe topology and switches are stressed as critical when combining many GPUs and high-speed NICs.

Cluster Design: New vs Used Hardware

  • Some argue used Mellanox/IB gear is cheap and reliable for labs.
  • Others reject used gear for a commercial GPU cloud, prioritizing vendor support contracts, uptime guarantees, and avoiding blame on unsupported hardware.

AMD MI300x vs Nvidia H100 and Software Stack

  • One operator is building a 128-GPU MI300x Ethernet cluster, positioning AMD as the “underdog” with strong vendor support and lower strategic risk than joining the Nvidia crowd.
  • ROCm is recognized as the CUDA-equivalent but widely viewed as immature; some ML users say that alone is a deal-breaker today.
  • Technical limitations (e.g., missing PCIe passthrough for multi-tenant VMs) currently block fine-grained GPU rental; container-based workarounds have drawbacks.

Datacenter Power and “Green” Claims

  • Multiple comments discuss “green” datacenters, hydro-heavy regions (PNW, Quebec, Iceland), and future solar+battery-powered DCs.
  • There is debate over whether buying “green” power actually reduces global CO₂, versus needing strong regulation.
  • Large AI clusters are estimated to consume on the order of tens to hundreds of MW; renewables plus large batteries are argued by some to be already cost-competitive at that scale.