Google's First Tensor Processing Unit: Architecture

Original Article ↗ Hacker News Discussion ↗

Google’s first Tensor Processing Unit (TPU) prompts debate over what “tensor” really means in machine learning, with many noting that both TPUs and GPUs ultimately reduce higher‑dimensional arrays to vector and matrix operations despite the marketing and borrowed math terminology. Commenters contrast TPU’s more specialized, compiler‑dependent architecture and limited availability with Nvidia’s general‑purpose GPUs, mature CUDA ecosystem, and broad hardware lineup, arguing this software and distribution advantage explains Nvidia’s dominance in AI compute. The thread also reflects on how Google, despite inventing core technologies like transformers and TPUs, has struggled to capture AI’s commercial upside due to internal caution, cloud‑only hardware strategy, and worries about disrupting its ad‑driven search business.

Meaning and Use of “Tensor”

  • Debate over why “tensor” is used when hardware mainly does vector/matrix ops.
  • Several comments say it’s largely branding and shorthand for n‑dimensional arrays; higher ranks often just come from batching (e.g., N,C,H,W for images).
  • Others stress that “tensor” in pure math/physics means a coordinate‑transforming multilinear object, not just an array; ML “tensors” are closer to generic multidimensional arrays.
  • Historical notes: the word’s meaning has drifted over a century; its use in ML as near‑synonym for “array” is described as recent and somewhat unfortunate but entrenched.
  • Some argue the ML use is acceptable within that context; others consider it misleading.

TPU Architecture and Programming Experience

  • TPU v1 described as having ~20 instructions; commenters question calling that “CISC,” debating what CISC/RISC even means here.
  • TPUs conceptually operate on matrices; n‑D tensors are flattened into 2D matmuls (e.g., 3×4×256 → 12×256) and mapped to fixed 2D “native sizes.”
  • One perspective: higher‑order tensor operations decompose into lower‑dimensional ones, so the hardware “supports tensors” via composition.
  • Practitioner reports: TPUs can be very fast for architectures they were designed for, but compilers can be fragile; new or unusual models may run poorly or even slower than CPUs.
  • GPUs are seen as more general, with mature low‑level access enabling custom kernels like FlashAttention.

Nvidia, CUDA, and Competing Accelerators

  • Nvidia’s key advantages cited: reserved fab capacity, deeply integrated software (CUDA, libraries), existing datacenter hardware, and customer relationships.
  • Many see CUDA’s optimized kernels and ecosystem as the main moat; alternatives (ROCm, Vulkan, Triton, vendor TPUs, AWS Trainium, Intel/Habana, AMD GPUs) exist but lag in adoption and tooling.
  • Some complain that CUDA and drivers are brittle; others respond that competitors’ stacks are worse.
  • View that AI demand and big‑tech self‑interest will eventually produce serious competition; counter‑view that replicating Nvidia’s software stack is an enormous, long‑term effort.

Google’s AI/TPU Strategy and Missed Opportunities

  • Recurrent question: how Google invented TPUs and the transformer, built strong chat models internally, yet allowed Nvidia and startups to capture most visible value.
  • Explanations offered:
    • Fear of safety, hallucinations, and regulatory backlash, especially in the EU.
    • Internal conservatism and reluctance to launch radical new products that might threaten search/ads.
    • Complacency and preference for incremental improvements vs disruptive bets.
  • Some argue it’s premature to say Google has “lost”; they can still embed AI across products, but matching search‑scale revenue is hard.
  • TPU business: calls for spinning out the team as a separate company; concern that TPUs are scarce even internally and mostly accessible via Google Cloud, limiting ecosystem growth.

Fabrication, Availability, and Other Hardware Notes

  • TPUs reportedly use slightly older process nodes than cutting‑edge GPUs; this is considered acceptable but less glamorous.
  • Proposal that Google build its own leading‑edge fab is widely dismissed as economically and technically infeasible given required capital, expertise, and volumes.
  • Apple’s M‑series chips are noted as strong for local inference due to unified memory and high bandwidth, but not positioned as competitors for large‑scale training like H100s.