Why do tree-based models still outperform deep learning on tabular data? (2022)

Original Article ↗ Hacker News Discussion ↗

Tree-based models like gradient-boosted trees and random forests still tend to beat deep neural networks on tabular data, largely because they handle small, noisy, heterogeneous datasets well and are easier to train, interpret, and deploy. Commenters highlight that deep learning shines on raw, high-dimensional data (images, text, audio) and in some large-scale tabular settings such as recommendation systems or when pretraining across many datasets, but often struggles to justify its complexity for typical business data from relational databases. There is growing interest in hybrid and differentiable tree architectures, as well as “tabular foundation models,” yet many practitioners report that well-tuned tree models like XGBoost and CatBoost remain the strongest baseline in practice.

Motivation for Deep Learning on Tabular Data

  • Some see little need: linear models and gradient-boosted trees (GBTs/GBMs, CatBoost, XGBoost) already work very well, are easier to interpret, and can overfit most datasets.
  • Others argue that in highly competitive domains (e.g., finance, large-scale recommendation, ads/search ranking), “good enough” isn’t sufficient; if NNs can add even small gains, they matter.
  • Another key motivation: trees are non‑differentiable, so tabular‑specific NN architectures would integrate better with other deep learning components in end‑to‑end systems.

Why Tree-Based Models Often Win on Tabular Data

  • Strong inductive bias for heterogeneous features, small datasets, extreme values, and clustered/non‑IID data.
  • Naturally handle non‑linear decision boundaries and complex interactions via splits.
  • More robust to distribution shift / nonstationarity in some views, because splits are threshold-based rather than additive.
  • Easier to train, deploy, debug, and explain for most business tabular problems.
  • Widely reported to dominate in competitions and large-scale industrial setups (search, ads, exabyte-scale tabular data).

Arguments for Neural Networks on Tabular Data

  • Some practitioners report NNs already outperform strong tree baselines on specific tabular problems, especially with complex relationships or very high-dimensional data.
  • Advantages cited:
    • Better for streaming/online learning; trees typically need retraining from scratch.
    • Potential for extrapolation beyond observed target ranges where trees are limited to averages of seen values (though this is debated).
    • Success in recommendation, ranking, and time-series–like tasks; multi-stage architectures can mitigate inference cost.
    • Ensembles of multiple NNs can beat single tree models, at the expense of training complexity.
  • Newer work (including “tabular foundation” and pre-trained tabular models) shows cases where pre-trained deep models can markedly outperform trees, especially when leveraging string features and transfer learning.

Hybrid and Alternative Directions

  • Interest in differentiable or neural-augmented trees (e.g., differentiable forests, tree+NN hybrids).
  • Attention/Transformer-style architectures across features and feature embeddings proposed to encode tabular inductive biases.
  • Some argue many tabular use cases are better served by relational databases and SQL; ML is more justified when the goal is prediction or discovering unknown structure.