Google's First Tensor Processing Unit: Architecture

Original Article ↗ Hacker News Discussion ↗

Meaning and Use of “Tensor”

  • Debate over why “tensor” is used when hardware mainly does vector/matrix ops.
  • Several comments say it’s largely branding and shorthand for n‑dimensional arrays; higher ranks often just come from batching (e.g., N,C,H,W for images).
  • Others stress that “tensor” in pure math/physics means a coordinate‑transforming multilinear object, not just an array; ML “tensors” are closer to generic multidimensional arrays.
  • Historical notes: the word’s meaning has drifted over a century; its use in ML as near‑synonym for “array” is described as recent and somewhat unfortunate but entrenched.
  • Some argue the ML use is acceptable within that context; others consider it misleading.

TPU Architecture and Programming Experience

  • TPU v1 described as having ~20 instructions; commenters question calling that “CISC,” debating what CISC/RISC even means here.
  • TPUs conceptually operate on matrices; n‑D tensors are flattened into 2D matmuls (e.g., 3×4×256 → 12×256) and mapped to fixed 2D “native sizes.”
  • One perspective: higher‑order tensor operations decompose into lower‑dimensional ones, so the hardware “supports tensors” via composition.
  • Practitioner reports: TPUs can be very fast for architectures they were designed for, but compilers can be fragile; new or unusual models may run poorly or even slower than CPUs.
  • GPUs are seen as more general, with mature low‑level access enabling custom kernels like FlashAttention.

Nvidia, CUDA, and Competing Accelerators

  • Nvidia’s key advantages cited: reserved fab capacity, deeply integrated software (CUDA, libraries), existing datacenter hardware, and customer relationships.
  • Many see CUDA’s optimized kernels and ecosystem as the main moat; alternatives (ROCm, Vulkan, Triton, vendor TPUs, AWS Trainium, Intel/Habana, AMD GPUs) exist but lag in adoption and tooling.
  • Some complain that CUDA and drivers are brittle; others respond that competitors’ stacks are worse.
  • View that AI demand and big‑tech self‑interest will eventually produce serious competition; counter‑view that replicating Nvidia’s software stack is an enormous, long‑term effort.

Google’s AI/TPU Strategy and Missed Opportunities

  • Recurrent question: how Google invented TPUs and the transformer, built strong chat models internally, yet allowed Nvidia and startups to capture most visible value.
  • Explanations offered:
    • Fear of safety, hallucinations, and regulatory backlash, especially in the EU.
    • Internal conservatism and reluctance to launch radical new products that might threaten search/ads.
    • Complacency and preference for incremental improvements vs disruptive bets.
  • Some argue it’s premature to say Google has “lost”; they can still embed AI across products, but matching search‑scale revenue is hard.
  • TPU business: calls for spinning out the team as a separate company; concern that TPUs are scarce even internally and mostly accessible via Google Cloud, limiting ecosystem growth.

Fabrication, Availability, and Other Hardware Notes

  • TPUs reportedly use slightly older process nodes than cutting‑edge GPUs; this is considered acceptable but less glamorous.
  • Proposal that Google build its own leading‑edge fab is widely dismissed as economically and technically infeasible given required capital, expertise, and volumes.
  • Apple’s M‑series chips are noted as strong for local inference due to unified memory and high bandwidth, but not positioned as competitors for large‑scale training like H100s.