Friends don't let friends export to CSV

Original Article ↗ Hacker News Discussion ↗

CSV’s status as the universal “export” format is challenged by engineers who argue it’s underspecified, fragile, and inefficient for large-scale data pipelines, pointing instead to formats like Parquet, SQLite, or line-delimited JSON. Others counter that CSV’s human readability and near-universal support—especially via Excel and similar tools—make it indispensable for business users and cross‑organization data exchange, despite its quirks with locales, encodings, and type ambiguity. The emerging consensus is to keep offering CSV where humans and spreadsheets are involved, but to favor more structured, schema-aware formats when shuttling large or critical datasets between systems.

Why CSV remains dominant

  • Universally supported: almost every business tool, database, and language can import/export CSV, often with no extra libraries.
  • Works for non-technical users: people know “export/import CSV” and “open in Excel/Sheets”; many workflows, FOIA responses, and legacy systems are built around it.
  • Easy to eyeball and hand-edit for small or medium files. This is critical for debugging integrations and “one broken file” incidents.
  • For modest data sizes (MBs to low GBs), performance is usually good enough, especially when zipped.

Practical problems with CSV

  • Underspecified “family” of formats: different delimiters (,, ;, \t, |), quoting rules, header presence, encodings (UTF‑8 vs legacy), decimal separators, date formats, and null representations.
  • Real‑world files often violate RFC 4180: inconsistent quoting, wrong escaping, comments, misaligned rows, nested CSV in fields. A single robust parser for “all CSVs” is effectively impossible.
  • Locale and Excel issues: Excel changes separator based on locale, guesses types, converts IDs to numbers/dates/scientific notation, strips leading zeros, and can silently corrupt data on save. LibreOffice is often reported as more sane.
  • Encoding and schema are out‑of‑band; consumers must guess types and formats, leading to subtle bugs.

Debate over “human readable”

  • Pro: Human readability is seen as a major virtue. You can inspect, diff, and surgically fix data without code. CSV is considered durable and future‑proof.
  • Con: Human readability encourages ad‑hoc, buggy producers (“just join with commas”), making interoperability worse. Some argue binary + good tools beats text + misuse.

Parquet and other alternatives

  • Parquet praised for data pipelines: columnar, typed, schema‑aware, efficient for large datasets and partial reads; compresses well (comparable to csv.gz).
  • Criticisms: more complex spec, spotty or painful library support in some languages/environments, version quirks, not directly usable in Excel, and not ideal for row‑streaming use cases.
  • Other suggested formats: SQLite files, Arrow, Avro, Protobuf/Cap’n Proto, JSON/NDJSON, TSV, XML with schema, CSVY (CSV + YAML metadata), and “Unicode/ASCII separated values” using dedicated delimiter characters.

When CSV is appropriate vs not

  • Widely agreed:
    • Keep CSV (or XLSX) for external stakeholders and ad‑hoc analysis in spreadsheets.
    • Consider Parquet/SQLite/Arrow/etc. for internal, large, or performance‑sensitive pipelines where you control both ends.
  • Core tension: CSV is technically fragile but socially entrenched. Many see replacing it at scale as unrealistic; others argue we should at least stop using it as the default for “serious” system‑to‑system integration.