Kolmogorov-Arnold Networks

Original Article ↗ Hacker News Discussion ↗

Kolmogorov-Arnold Networks (KANs) are a newly popular neural network architecture that replace traditional neuron activations with learnable spline functions on each weight, promising high accuracy and improved interpretability for low- to moderate-dimensional scientific and regression tasks. Commenters highlight intriguing results on symbolic formula recovery and function fitting, while questioning how well KANs scale to high-dimensional data, deep architectures, and workloads like image classification or large language models, especially given GPU-unfriendly branching and training instability. There is broad enthusiasm for the conceptual novelty and links to prior ideas (splines, GAMs, symbolic regression), tempered by calls for larger benchmarks and more efficient implementations before KANs can challenge mainstream architectures such as MLPs and transformers.

Overall reaction

  • Many find Kolmogorov–Arnold Networks (KANs) refreshing compared to incremental transformer tweaks, especially for their mathematical grounding and interpretability.
  • Enthusiasm is tempered by concern that current results are mostly on low-dimensional, toy or physics-style problems.

Architecture & theory

  • KANs place learnable univariate functions (splines or other bases) on edges and use simple summation at nodes.
  • They operationalize the Kolmogorov–Arnold representation (continuous multivariate functions as sums of univariate functions), with depth and width generalized beyond the classic 2-layer construction.
  • Some note similarity to existing ideas: GAMs, symbolic regression, finite elements, decision trees (loosely), Bayesian networks, GNN-like edges.

Training behavior and optimization

  • Reference implementation uses LBFGS and is described as slow and numerically delicate.
  • Deep stacks or naive replacements of MLP/attention layers with KANs can lead to diverging loss; training appears less stable than ReLU MLPs.
  • Overfitting can be severe on small datasets; more data plus “grid-annealing” (gradually increasing spline grid resolution) helps.

Scaling, GPUs, and hardware

  • Several question GPU-friendliness. Per-edge, per-weight activations induce branching and poor matmul utilization; some report CUDA being slower than CPU.
  • Suggestions include custom kernels, basis precomputation, or transform-based approximations (Fourier/DCT), but efficient large-scale implementations are still unclear.

Applications, benchmarks, and limitations

  • Current demos mostly use ≤100 input dimensions; commenters repeatedly request MNIST or larger benchmarks and note that business problems often work fine with trees.
  • KANs seem well-suited to regression with interpretable structure or when a compact closed-form is desired, less obviously to large-scale classification or language modeling.
  • Claims of reduced catastrophic forgetting are intriguing but not yet backed by large, public benchmarks.

Interpretability, symbolic regression, and related methods

  • A major attraction is extracting human-readable formulas from trained networks via pruning and operator libraries.
  • Commenters connect KANs to symbolic regression, grammatical evolution, and prior “distill laws from data” work; KANs can be seen as a more gradient-based, structured variant.

Experiments and alternative implementations

  • Multiple independent PyTorch variants appear (Fourier-based, efficient KANs, KAN-GPT), plus experiments with alternative bases (radial basis functions).
  • Some find good performance on synthetic classification tasks after substantial tuning, but scaling to deep nets and large vocabularies remains an open question.