Kolmogorov-Arnold Networks

Original Article ↗ Hacker News Discussion ↗

Overall reaction

  • Many find Kolmogorov–Arnold Networks (KANs) refreshing compared to incremental transformer tweaks, especially for their mathematical grounding and interpretability.
  • Enthusiasm is tempered by concern that current results are mostly on low-dimensional, toy or physics-style problems.

Architecture & theory

  • KANs place learnable univariate functions (splines or other bases) on edges and use simple summation at nodes.
  • They operationalize the Kolmogorov–Arnold representation (continuous multivariate functions as sums of univariate functions), with depth and width generalized beyond the classic 2-layer construction.
  • Some note similarity to existing ideas: GAMs, symbolic regression, finite elements, decision trees (loosely), Bayesian networks, GNN-like edges.

Training behavior and optimization

  • Reference implementation uses LBFGS and is described as slow and numerically delicate.
  • Deep stacks or naive replacements of MLP/attention layers with KANs can lead to diverging loss; training appears less stable than ReLU MLPs.
  • Overfitting can be severe on small datasets; more data plus “grid-annealing” (gradually increasing spline grid resolution) helps.

Scaling, GPUs, and hardware

  • Several question GPU-friendliness. Per-edge, per-weight activations induce branching and poor matmul utilization; some report CUDA being slower than CPU.
  • Suggestions include custom kernels, basis precomputation, or transform-based approximations (Fourier/DCT), but efficient large-scale implementations are still unclear.

Applications, benchmarks, and limitations

  • Current demos mostly use ≤100 input dimensions; commenters repeatedly request MNIST or larger benchmarks and note that business problems often work fine with trees.
  • KANs seem well-suited to regression with interpretable structure or when a compact closed-form is desired, less obviously to large-scale classification or language modeling.
  • Claims of reduced catastrophic forgetting are intriguing but not yet backed by large, public benchmarks.

Interpretability, symbolic regression, and related methods

  • A major attraction is extracting human-readable formulas from trained networks via pruning and operator libraries.
  • Commenters connect KANs to symbolic regression, grammatical evolution, and prior “distill laws from data” work; KANs can be seen as a more gradient-based, structured variant.

Experiments and alternative implementations

  • Multiple independent PyTorch variants appear (Fourier-based, efficient KANs, KAN-GPT), plus experiments with alternative bases (radial basis functions).
  • Some find good performance on synthetic classification tasks after substantial tuning, but scaling to deep nets and large vocabularies remains an open question.