Pandas Should Go Extinct

Original Article ↗ Hacker News Discussion ↗

Claims that Python’s pandas library should be phased out in favor of Polars and DuckDB spark a broader examination of data tooling, performance, and ergonomics. Many argue that newer columnar and SQL-oriented tools offer clearer APIs, fewer pitfalls, and far better scalability, even for modest datasets, while others note pandas remains “good enough” for most analysts given its ecosystem, documentation, and tight integration with plotting and ML libraries. The exchange highlights tensions between legacy tooling and modern alternatives, the often-overstated prevalence of “big data,” and the evolving role of data science in an era dominated by AI and improved database engines.

Adoption of Polars and DuckDB vs Pandas

  • Several commenters now default to Polars or DuckDB, citing clearer syntax, better performance, and fewer “footguns.”
  • Pandas is seen as deeply entrenched in teams and training materials, making migration politically and culturally hard.
  • Some still prefer pandas for quick exploratory work, visualization, and because they know it best.

Performance and “big data” considerations

  • Many note most real-world pandas use is on small-to-medium data; performance and memory problems tend to appear only beyond tens of GBs.
  • Others report measurable gains from Polars/DuckDB even for small ETL pipelines and Excel ingestion (seconds vs minutes).
  • One experience suggests Polars’ “lazy” APIs sometimes materialize data unexpectedly and blow up memory, while DuckDB behaves more predictably.
  • Another counterpoint: Polars can actually be slower on small, simple operations (e.g., small sorts) unless tuned (e.g., single-threaded). Profiling is emphasized.

API design, ergonomics, and maintainability

  • Polars and DuckDB APIs are described as easier to reason about than pandas, especially for group-bys, window functions, and declarative column expressions (col-style).
  • Critiques of pandas: unintuitive, brittle pipelines where small question changes require large code rewrites; heavy lambda use can hurt vectorization.
  • Defenders argue that “properly written” pandas looks similar to Polars/tidyverse and that pandas 3’s pd.col narrows the gap.

Ecosystem integration and tooling gaps

  • Tight integration with scikit-learn, matplotlib, and broader ML tooling keeps many on pandas.
  • Polars interop via to_pandas() / from_pandas() / to_numpy() is seen as workable, but edge cases still occur across the ML ecosystem.
  • There is still significantly more demand for pandas training than Polars training.

Role of SQL, Dask, and other alternatives

  • Some advocate pushing as much work as possible into databases or DuckDB/ClickHouse, keeping Python as “glue.”
  • Dask is mentioned as under-discussed in the article: used successfully for 10TB+ workloads, though I/O and hardware configuration often dominate performance.
  • Other tools referenced: Modin, cuDF, chdb DataStore (pandas API on ClickHouse engine), Ibis for a dplyr-like API, and Excel/JSON/CSV/Parquet/S3 integration concerns.

Reactions to the article and title

  • The title is seen as catchy and clickbaity; some enjoyed the bait-and-switch, others felt it was needlessly harsh on pandas.
  • Initial link mix-ups between two posts caused confusion.
  • Several stress that pandas “served us well” and remains a valid, evolving tool, even if Polars/DuckDB are often technically superior for new projects.