GPUs Go Brrr
GPUs are increasingly being treated as highly specialized AI machines, with tools like Stanford’s ThunderKittens pushing NVIDIA H100 hardware close to its theoretical limits by optimizing low-level matrix operations and tensor cores. Commenters weigh how far this GPU-centric path can go versus purpose-built AI accelerators (NPUs, TPUs, analog or ASIC designs), and debate vendor lock‑in, VRAM constraints, and the dominance of NVIDIA’s software ecosystem. Many see better abstractions, open tooling, and competing hardware (from AMD, Apple, Groq and others) as crucial for spreading advanced AI workloads beyond large data centers to prosumer and mobile devices.
ThunderKittens, H100, and matrix-centric “AI hardware”
- Many see the post and ThunderKittens as a clear, fun illustration of how to hit H100 tensor cores properly; some find the meme-heavy style off‑putting or confusing.
- The core idea that “AI wants 16×16 tiles” and that tensor cores dominate available FLOPs resonates; several note that not using tensor cores wastes ~94% of H100’s theoretical compute.
- There’s debate over how much further specialized “AI ASICs” can go beyond modern GPUs. Some argue GPUs have already evolved into AI machines; others think trimming graphics baggage and precision could still yield big gains.
GPUs vs NPUs / TPUs / custom accelerators
- Commenters compare NVIDIA tensor cores with Apple NPUs, Google TPUs, Groq, AMD Versal/XDNA, and various NPUs.
- View that NPUs on consumer devices shine for small, low-power tasks but are underpowered for LLMs/large transformers; GPUs still win for big models.
- Some think future will bring simpler, cheaper AI-only chips; others cite NVIDIA research claiming limited benefit from further specialization.
Memory, bandwidth, VRAM, and form factors
- Strong focus on memory bandwidth vs latency; GPUs hide latency but are bottlenecked by bandwidth and VRAM capacity.
- Many lament lack of high‑VRAM consumer cards and absence of expandable VRAM; several argue it’s economically and strategically undesirable for vendors.
- Unified memory (as in Apple silicon or potential advanced packaging) is seen as attractive for large models but not yet standard.
NVIDIA dominance, docs, and competition
- Some accuse NVIDIA of protecting its moat with incomplete or misleading documentation; others see ordinary market segmentation (better docs for “partners”).
- There's a camp calling for NVIDIA to be “broken up”; a larger counter‑camp credits long‑term investment and superior software (CUDA, tooling, PyTorch support) rather than anti‑competitive barriers.
- AMD is viewed as promising but still behind in software and ecosystem, though recent ROCm progress and MI300x interest are noted.
Local vs cloud AI and consumer devices
- Split between those who want strong on‑device models (privacy, offline speech/vision, accessibility) and those who prefer cloud “AI as a service” to save battery/thermals.
- Apple is seen as likely to blend modest on-device AI with iCloud-based services, trading some privacy positioning for better UX.
Beyond current GPUs: neuromorphic and analog ideas
- Some argue real AGI may require insights from neuroscience/psychology; others think current ML is largely decoupled from brain science.
- Analog and neuromorphic approaches are discussed as active but immature: major challenges in precision, noise accumulation, manufacturing, and debugging.