Polars 2.0

Original Article ↗ Hacker News Discussion ↗

Polars 2.0, a Rust-based dataframe engine, is being received as a major upgrade that adds first-class SQL support, improves performance on analytical benchmarks, and introduces better scaling and out-of-core capabilities. Many users report migrating from pandas and even Spark-style pipelines to Polars (often alongside DuckDB), citing its cleaner API, speed on large datasets, and suitability for production workloads ranging from quantitative trading to weather analytics. At the same time, commenters note gaps such as immature geospatial tooling and caution that benchmark results against DuckDB and DataFusion should be interpreted as indicative optimizations rather than absolute performance rankings.

Release timing & positioning

  • Some note the coincidence with DuckDB’s upcoming 2.0; Polars contributors say 2.0 has been planned for a long time, with work on the branch since mid‑2026.
  • Several comments frame Polars 2.0 as directly competing with DuckDB and DataFusion, especially via SQL support and TPC-H/TPC-DS benchmarks.

Performance, streaming, and benchmarks

  • Multiple users report large speedups over pandas; one cites a “1 Billion Row” benchmark where Polars and DuckDB are ~50x faster than pandas.
  • Streaming/out-of-core support is highlighted as a major improvement, addressing cases where pandas/polars previously ran out of memory.
  • Polars maintainers acknowledge overhead issues on very high core counts (e.g., 192 threads) due to join partitioning, and plan further optimization.
  • Some report DataFusion beating Polars for their workloads; others see DuckDB slower than both.
  • Benchmarking experts warn against reading vendor benchmarks as absolute rankings; TPC-H/DS are useful but not fully representative of modern, especially semi-structured, workloads.

Polars vs Pandas vs DuckDB (and others)

  • Many say Polars is a near drop-in replacement for pandas in “99.9%” of cases, with better performance, scaling, and a cleaner API.
  • Noted gaps: geospatial (GeoPandas is mature; GeoPolars is still early), some row-wise/in-place operations, richer ecosystem/interoperability (plotting, stats).
  • DuckDB is praised for SQL-centric workflows, low memory usage, geospatial extension, and working as an in-process analytical database.
  • Some prefer SQL/DuckDB for complex cleaning (one query vs many lines of dataframe code), while others like Polars’ dataframe API over SQL.

Features, SQL, and missing pieces

  • SQL support in Polars is seen as a big step; however, certain operations (e.g., asof_join) are not yet available via SQL even though they exist in the core API.
  • Questions are raised about timestamp precision; another commenter states nanosecond Datetime("ns") works when schema is specified.
  • Initial complaints about awkward string concatenation are countered with examples showing Pandas-like syntax is possible.

Ecosystem, interoperability, and use cases

  • Narwhals is mentioned as a dataframe-agnostic layer to abstract over pandas/polars/etc.
  • Teams report moving from pandas/R stacks to Polars + DuckDB.
  • Real-world uses include quant trading bots, large-scale parcel data processing, and precomputing billions of weather scores.

Concerns: memory, threading, and usability

  • Some experience higher memory usage and OOMs due to heavy multithreading; reducing POLARS_MAX_THREADS sometimes improves both stability and speed.
  • A few commenters stick with pandas for small, science-oriented workflows because downstream libraries expect pandas/NumPy.