Llm.c – LLM training in simple, pure C/CUDA
A minimalist large language model training implementation in ~1000 lines of C has renewed interest in doing deep learning without heavyweight Python frameworks like PyTorch. Commenters weigh the tradeoffs between Python’s convenience and ecosystem versus the performance, simplicity, and smaller dependency footprint of low-level C/CUDA code, while noting that GPU stacks like CUDA still dominate and remain huge and complex. Many see this project less as a practical PyTorch replacement and more as an educational tool that demystifies how transformers and training actually work under the hood.
Project scope and intent
- Repo is a ~1000-line C implementation of transformer training, currently CPU/OpenMP-based; CUDA is planned but not yet present.
- Many see it as a minimal, didactic reference showing that LLM training internals are conceptually simple compared to large frameworks.
- Others note that while the core algorithm fits in little code, production systems need far more functionality and engineering.
Python, overhead, and frameworks
- Debate over Python’s performance: core tensor ops are in optimized low-level code, so runtime/memory overhead is often modest.
- Counterpoints: hot Python loops and suboptimal data structures can dominate runtime; “using the wrong primitive” can yield huge slowdowns.
- Some argue Python’s container variety invites performance traps; others say that’s a programmer, not language, problem.
- Larger criticism targets ecosystem complexity and dependency bloat (PyTorch, CPython) versus small, focused C code.
CUDA, GPU stacks, and alternatives
- CUDA SDK and associated libraries are seen as huge and complex (multi‑GB installs, very large shared objects).
- Some emphasize that CUDA size is also a burden for Python stacks; GPU complexity is largely independent of language.
- CUDA remains dominant in speed and tooling; alternatives (AMD ROCm, TPUs, Vulkan-based projects, Rust frameworks, etc.) exist but are less mature or harder to use.
- Several comments describe frustrating experiences with AMD drivers and ecosystem stability.
Hardware and VRAM discussions
- Desire for expandable GPU memory (e.g., SO‑DIMMs) to reduce VRAM constraints for LLMs.
- Others explain technical and economic reasons against modular GPU memory: latency, bandwidth, power, cooling, and product segmentation.
Educational value and resources
- Many view the project as an excellent learning tool that demystifies LLM training.
- Multiple replies point to associated tutorials and video series that build similar models step by step in higher-level languages, then in C.
Transformers and other domains
- Discussion notes that transformers are generic “array in, array out” modules; with appropriate tokenization they can handle time series, images, and audio.
- References are given to time-series transformer work, alongside skepticism that these always beat strong classical baselines like gradient boosting.