Should you ditch Spark for DuckDB or Polars?

Original Article ↗ Hacker News Discussion ↗

A growing number of data engineers are questioning whether they really need Apache Spark’s distributed complexity when tools like DuckDB and Polars can handle hundreds of gigabytes — and sometimes terabytes — on a single, well-provisioned machine. Commenters weigh trade-offs in performance, cost, and developer experience, noting that many organizations overestimate their “big data” needs and could benefit from simpler, SQL‑centric or dataframe‑based workflows, while truly massive or heavily concurrent workloads still justify Spark or similar systems. The conversation also touches on file formats, catalog and orchestration gaps, integration with cloud platforms, and how LLM-assisted coding and vendor ecosystems influence tool choices.

Spark vs. Single-Node Scale

  • Several commenters argue most “big data” workloads don’t justify Spark. 100GB–a few TB is often manageable on a single large machine with fast NVMe.
  • Others note real large-scale cases: 1TB/day FX or clickstream data, 40TB+ OLTP, and 100+ concurrent analysts on shared datasets. In such contexts, distributed systems like Spark are still warranted.
  • Single-node limits are not just disk size but also CPU, RAM, and bandwidth, plus redundancy concerns. Some treat compute as ephemeral and rebuild from object storage if a node dies.

DuckDB: Strengths, Limits, and Usage Patterns

  • Many report that DuckDB “just works,” especially due to out-of-core execution and good performance on TB-scale workloads.
  • Some have fully migrated warehouses to DuckDB’s native format for cost and speed, keeping Parquet only for interoperability. Others still prefer Parquet/Delta as a stable interchange format.
  • Concerns: lack of horizontal scaling, weaker catalog features vs. cloud warehouses, less mature optimizer (more manual query tuning), and uncertain best practices for very large, constantly-updated native files.
  • Integration with Spark and Arrow is seen as a big plus; some advocate combining DuckDB for most workloads with Spark/Databricks only when scale demands it.

Polars: Capabilities and Pain Points

  • Praised for complex transformations and a strong plugin story (e.g., Rust-based geospatial extensions with major speedups).
  • Criticized for frequent OOM on large workloads compared to DuckDB and for API instability in earlier phases.
  • SQL support exists but is not as central as in Spark/DuckDB; typically used in lazy DataFrame style.

Data Formats, Catalogs, and Alternatives

  • Debate over Delta vs. open formats: Delta is open but tightly tied to Spark/Databricks, with features (e.g., deletion vectors) that can break OSS compatibility. Iceberg is discussed as a unifying, multi-engine layer.
  • Multiple alternatives surface: Ray (general distributed compute), Daft (on Ray), ClickHouse (fast but catalog and OOM concerns), Lake Sail, DataFusion, LanceDB, and translation layers like Fugue and Ibis.

Orchestration, DX, and LLMs

  • Spark/Databricks win on built-in streaming, autoloading, checkpointing, and workflow management; with DuckDB/Polars you typically add Airflow/Kestra/dbt-like tools.
  • Some emphasize that migration/ops complexity can outweigh performance gains.
  • Commenters note LLM assistants are currently better with Pandas/Spark than with newer APIs like Polars or Ibis, which may slow adoption.