Understanding Parquet, Iceberg and Data Lakehouses

Original Article ↗ Hacker News Discussion ↗

Modern data teams are increasingly standardizing on open table formats like Apache Iceberg, Delta Lake, and Hudi atop Parquet files in object storage, aiming to decouple storage from query engines and avoid vendor lock-in. Commenters weigh trade-offs between Iceberg and Delta in terms of openness, ecosystem support (especially beyond the JVM), and real-world performance, noting how major players like Snowflake, Microsoft, and Google are lining up behind different options. Beneath the format details, many argue that lakehouses largely repackage classic data-warehouse ideas (star schemas, ETL/ELT, governance) on cheaper cloud storage, and that modeling and data quality still matter far more than the specific file or metadata format chosen.

What the New Table Formats Do

  • Parquet widely seen as the de facto columnar storage for data lakes; Iceberg/Delta/Hudi add table-level metadata, schema evolution, partitioning, and ACID semantics on top.
  • This decouples components: e.g., S3 for storage, Parquet for data, Iceberg/Delta for metadata, DuckDB/Spark/Trino/Snowflake/BigQuery as engines.

Openness and Governance (Iceberg vs Delta vs Hudi)

  • Iceberg is viewed as a cleaner, well-specified open standard; Delta’s spec is seen as complex and tightly coupled to Databricks’ implementation.
  • Some argue Delta is only “nominally” open: spec changes driven by one vendor, some features held back from open source.
  • Counterpoint: others say Delta has been open enough in practice for years, and large players like Microsoft are betting heavily on it.
  • Several note that Snowflake and BigQuery are adding Iceberg (not Delta), interpreted as a signal of Iceberg’s neutrality.

Tooling and Language Ecosystem

  • Historically, Iceberg was Spark/Hadoop-centric; non-JVM support (PyIceberg, DuckDB extension) is arriving but still maturing.
  • Delta has delta-rs for Python/Rust and is perceived as easier outside the JVM today, though its origin and governance are debated.
  • Trino has removed Hadoop/Hive dependencies; many see the ecosystem gradually moving beyond old big-data stacks.

Data Lakes, Warehouses, and Lakehouses

  • Strong theme: formats are implementation details; real value comes from modeling (e.g., star schemas), ETL/ELT, and BI/reporting.
  • Data lakes alone often turn into “swamps” with lots of raw data and few insights; lakehouses reintroduce table-like structure and transactions on top of cheap object storage.
  • Some argue most enterprises don’t need lakes at all, given modest data sizes and existing relational systems.

Performance, Cost, and Portability

  • Table formats on object storage promise cheaper, vendor-neutral storage and the ability to swap query engines with less lock-in.
  • Several highlight that querying lakes/lakehouses is generally slower than purpose-built columnar databases; acceptable for many analytics, problematic for interactive BI.
  • Vendors claim they aim to make open table formats perform close to native storage, but whether they will abandon proprietary internals is questioned.

File Formats and Practical Issues

  • Feather vs Parquet: many prefer Parquet as more future-proof and interoperable; using Feather can break some Python lakehouse stacks.
  • CSV is widely criticized as brittle; Parquet adoption is seen as a big practical win.
  • Specialized formats like Lance and Zarr are mentioned for low-latency or multidimensional use cases, but still niche relative to Parquet.