Performance per dollar is getting faster and cheaper

AMD vs. Nvidia: Performance, Power, and Adoption

• Request for explicit performance-per-watt metrics; outside the US, electricity is relatively expensive and Nvidia supply is constrained.
• AMD MI355X cited at ~1,400W vs. Nvidia B200 at ~1,200W. One commenter says AMD is typically 20–60% worse on tokens/s/watt, but MI355X has ~50% more memory, complicating comparisons.
• Some see AMD as a fallback when Nvidia can’t fill orders; others argue large-scale operators still bend over backwards (loans, vendor financing) to get Nvidia rather than switch ecosystems.
• Mixed views on AMD’s trajectory: history of disappointment, but more experimentation now. AMD already widely used in consoles and EPYC CPU servers.
• Question raised whether MI355X is even rentable per hour yet; availability is unclear.

Power, Cooling, and Local Impact

• Example: a DGX B200 (~14 kW) over 8 years can consume ~1 GWh; in high-cost regions this is notable but still small compared to hardware capex.
• Main constraint is often grid capacity and power delivery, not energy price.
• Cooling adds ~10–20% overhead in some estimates, but high-density AI racks (30–50 kW) demand advanced liquid cooling and raise noise, weight, and infrastructure concerns.
• Dispute over datacenter externalities: some say they’re comparable to other large buildings; others highlight noise, vibration, pollution, and weak or selectively enforced regulations.
• Several argue regulations should target impacts (noise, emissions, water, etc.) rather than “data centers” by name.

Benchmarks, Throughput, and Quantization

• Clarification that 2,600 tokens/s is node aggregate; single-stream is ~213 tok/s when unsaturated, closer to ~30 tok/s per stream at full load. Calls for clearer per-user metrics.
• Debate on memory-bandwidth vs. compute bottlenecks: high-batch inference is often compute-bound.
• Heavy skepticism of FP4/MXFP4 “lossless” claims. Reported accuracy drops of 2–4% are seen as significant and can subjectively “lobotomize” models.
• MI355X can run FP6 at FP4 speed; some suggest MXFP6 could be nearly lossless with close-to-FP4 performance if workloads are compute-bound.
• Concern that some providers quantize already low-precision models or use aggressive post-training quantization mainly to chase Blackwell-like economics.

Economics and Pricing

• A provider reports ~40% gross margins, with utilization as the dominant factor.
• Despite hardware perf/$ gains, tokens-per-dollar are perceived as rising due to high demand, limited top-tier GPUs, and willingness to pay for “smarter” models; some suspect future price hikes after initial subsidization.

Meta and Framing

• Multiple critiques of the article/headline: “performance per dollar is getting faster,” “2x cheaper,” and omission of quantization level are seen as imprecise or misleading.
• Some contextualize current gains as part of a century-long exponential improvement in compute performance per dollar.