FireDucks: Pandas but Faster

Original Article ↗ Hacker News Discussion ↗

A new Python library called FireDucks promises up to 100x speedups over pandas while keeping near drop‑in API compatibility, attracting interest from teams with large existing pandas codebases. Commenters compare it to alternatives like Polars, DuckDB, and Ibis, often valuing Polars’ cleaner, Arrow-based API over raw speed, and highlighting pandas’ awkward but deeply entrenched interface—especially for time series work. Many are wary of FireDucks’ closed-source C++ core, BSD-licensed binary-only distribution, Linux-only support, and stated plan to commercialize, arguing that this makes it risky for long-term or production use despite impressive benchmark claims.

Scope and Compatibility

  • Marketed as a near drop‑in, “100% compatible” Pandas replacement; documentation admits it is not fully compatible but “pretty close.”
  • Targeted at single‑node workloads where Pandas is already used; not intended to replace distributed engines like Spark for very large jobs.
  • Currently Linux‑only, which limits applicability for some users.

Performance & Internals

  • Reported speedups come from multithreading, reimplementing core Pandas operations (e.g., dropna) in C++, lazy evaluation, and an MLIR/LLVM-based JIT that prunes unused work.
  • Some users impressed by large real‑world speedups (minutes → seconds) with zero code changes.
  • Others question claims of 100x over Pandas and 20–30% over Polars/DuckDB/ClickHouse; one user reran TPCH benchmarks and found Polars dramatically faster while FireDucks crashed, leading them to distrust the published numbers.
  • It’s unclear how it beats other modern engines that already use similar techniques (multithreading, Arrow, query optimization).

Licensing, Source, and Commercialization

  • Package is on PyPI under a 3‑clause BSD license, but the core C++ “magic” is only shipped as a binary; source is not available.
  • This creates confusion: legally open‑source binaries but practically closed source.
  • Several commenters see this as misleading or a “binary blob under an OSS license,” and worry about future lock‑in.
  • Official docs state current beta is free, with explicit intent to commercialize later; some view this as a “trap” for teams that adopt it before pricing is known.
  • Many say they will not adopt without C++ source, especially in finance/quant where inspectability is critical.

Pandas, Polars, and Alternatives

  • FireDucks’ main appeal: speedups for existing Pandas codebases without refactoring.
  • Polars is widely praised for a cleaner, more regular API and Arrow integration, but lacks some Pandas time‑series ergonomics; converting large Pandas codebases is non‑trivial.
  • Several users prefer sticking with Polars, DuckDB, DataFusion, or Ibis (often citing openness and Arrow-based interoperability).
  • Many complaints focus on Pandas’ clunky, footgun‑prone API rather than raw speed; some want a “better Pandas” API more than a faster backend.