Google TPU v5p beats Nvidia H100

Original Article ↗ Hacker News Discussion ↗

Google’s new TPU v5p accelerators are being weighed against Nvidia’s H100 and GH200 chips, with commenters probing not just raw specs but memory, bandwidth, power efficiency, and total cost of ownership at cloud scale. Many argue that Nvidia’s real advantage is its CUDA software ecosystem and broad availability, while TPUs remain tied to Google Cloud with uneven framework support—especially for PyTorch—creating significant switching costs. Others note emerging alternatives like AMD, Intel Gaudi, and custom cloud chips, but question whether any can gain traction without matching Nvidia’s mature tooling and widespread compatibility, particularly for cutting‑edge training workloads.

Hardware comparisons (TPU v5p vs Nvidia H100/GH200)

  • Several comments argue Google is “one generation behind,” noting TPU v5p is compared to H100 while Nvidia is already announcing GH200 and B100.
  • Raw specs disputes:
    • TPU v5p: 95 GB HBM per chip, ~2.7 TB/s memory bandwidth, 4,800 Gbps interconnect per chip, 3D torus topology.
    • GH200: contested figures; some claim 282 GB HBM3e per “superchip,” others clarify datasheets show 96–144 GB per GPU and that 282 GB refers to dual-GPU configs.
  • One engineer with apparent TPU ops experience gives pod details: 4 chips per host, 380 GB HBM per host, “cubes” of 64 TPUs (~6 TB HBM, ~29 PFLOPs bfloat16).
  • Consensus: GH200 likely beats TPU v5p per chip, but not by the huge margins suggested in some misquoted specs.

Cost, power, and TCO

  • Multiple comments stress performance-per-watt and total cost of ownership (TCO) matter more than peak FLOPs.
  • TPU pods are described as power-efficient vs GPUs due to lower TDP and torus interconnect needing fewer, less power-hungry links.
  • One user claims H100 delivers ~650 TFLOPs FP8, TPU v5p ~400 TFLOPs INT8, with v5p roughly similar to Intel Gaudi2 and cheaper than H100 on price/perf.
  • Google’s public “2.1x value-for-money” figure is clarified as TPU v5p vs TPU v4, not vs H100.

CUDA, software ecosystem, and lock-in

  • Large subthread argues Nvidia’s real moat is CUDA and its mature tooling, not raw hardware.
  • Many ML libraries and custom kernels (e.g., FlashAttention, DeepSpeed) are CUDA-first; non-CUDA backends typically lag.
  • Some note PyTorch, TensorFlow, and JAX can target TPUs (via XLA), but others say Pytorch-on-TPU is brittle and slower, with missing ops and data-loading issues.
  • General view: for cutting-edge research and community code, CUDA remains the path of least resistance.

Cloud-only access and market role of TPUs

  • TPUs are only available via Google Cloud; you cannot realistically buy and self-host them.
  • Several see this as limiting adoption, especially compared to Nvidia hardware which is widely sold.
  • Some argue TPUs were built primarily to reduce Google’s dependence on external vendors and serve internal workloads, not to be a standalone profit center.
  • This creates a dual lock-in concern: trading Nvidia/CUDA lock-in for Google Cloud/TPU lock-in.

Developer experience and programmability

  • Opinions diverge: some say using TPUs through TensorFlow/JAX is “painless”; others report version fragmentation across GCP services, unsupported ops (e.g., sparse tensors), and heavy refactoring needs.
  • Non-ML workloads: commenters say TPUs are deeply specialized; to use them you must target XLA/HLO. JAX is suggested for experimentation, but using TPUs for things like ray tracing is seen as niche.

Competing accelerators and future landscape

  • Other actors mentioned: AMD MI300, Intel Gaudi2/3, Cerebras, Tesla Dojo, Microsoft’s Maia, plus Apple and various inference ASICs.
  • Some believe Nvidia’s dominance may erode on inference as common model architectures standardize and non-CUDA stacks mature.
  • Others argue training will remain the main strategic bottleneck, where Nvidia’s ecosystem and interconnects keep it dominant for years.