Pre-Release of Polars 2.0

Original Article ↗ Hacker News Discussion ↗

Polars 2.0, a high-performance DataFrame library for Python and Rust, is preparing a major release that focuses less on new features and more on cleaning up old design decisions, tightening defaults, and enabling a new streaming-style execution engine that is more cache- and memory-efficient. Commenters compare Polars to pandas, SQL, DuckDB, and dplyr, debating syntax ergonomics but largely praising Polars for its speed, stricter typing, and production stability. Some concerns center on versioning consistency and defaults like non-deterministic row ordering, which can affect scientific workflows, but many see the changes as a step toward clearer, more explicit data pipelines.

Versioning philosophy and Polars 2.0

  • Many commenters appreciate that 2.0 is a “boring” release focused on breaking changes and cleaning up old decisions, not flashy features.
  • There is debate about Polars’ adherence to semantic versioning:
    • Some say minor versions already change/remove features often enough that version numbers “should be much higher.”
    • Others note that deprecations without breakage are semver-compliant, but Hyrum’s Law means any behavior change can break someone.
  • Some see 2.0 as primarily enabling future features and stricter defaults, not a feature release itself.
  • A few are annoyed that each major release feels like the API they memorized was “just a suggestion.”

Polars vs Pandas, SQL, and R/dplyr

  • Strong enthusiasm for Polars as a replacement for Pandas: faster, multithreaded, no index, more consistent, safer typing, better for production pipelines.
  • Some highlight the power of the lazy/expression API for composable abstractions and complex business logic.
  • Syntax is contentious:
    • Some find Polars and Pandas both less readable than SQL.
    • Others find SQL elegant for simple queries but messy once string-building, logic, and reuse are needed.
    • R/dplyr is praised as more ergonomic and beginner-friendly; Polars is seen as approaching that but still more verbose.
  • A few note you can query Polars with SQL, but lack of parameterized queries raises injection concerns.

Execution engine and “streaming”

  • “Streaming engine” here means a batched, in-cache, graph-based execution model, not infinite streaming.
  • It processes data in morsels that fit CPU cache, improving throughput and reducing memory needs; falls back to in-memory engine when needed.
  • Upcoming out-of-core work may move batches to disk when memory is exhausted.

Ordering, determinism, and maintain_order

  • Changing the default to maintain_order=False is controversial:
    • Supporters argue implicit ordering is a dangerous footgun and explicit ordering is better design, analogous to SQL.
    • Critics, especially from scientific workflows, worry about non-deterministic row order causing subtle, hard-to-detect bugs and making exploratory work more frustrating.
    • Some note that operations like group_by/join already produce order dependent on parallelization and hashing.

Production stability, typing, and ecosystem

  • Polars is praised for stricter typing, explicitness, and better production stability than Pandas, especially with messy data.
  • Some advocate prototyping in Python + Polars, then porting to Rust for stronger compile-time guarantees.
  • Static typing and linters help, but several argue Python’s type system still allows many runtime errors.
  • DuckDB, ClickHouse (clickhouse-local, chDB), dbt, and SQLAlchemy are discussed as adjacent or alternative tools, with differing views on ergonomics and scope.