Friends don't let friends export to CSV

Original Article ↗ Hacker News Discussion ↗

Why CSV remains dominant

  • Universally supported: almost every business tool, database, and language can import/export CSV, often with no extra libraries.
  • Works for non-technical users: people know “export/import CSV” and “open in Excel/Sheets”; many workflows, FOIA responses, and legacy systems are built around it.
  • Easy to eyeball and hand-edit for small or medium files. This is critical for debugging integrations and “one broken file” incidents.
  • For modest data sizes (MBs to low GBs), performance is usually good enough, especially when zipped.

Practical problems with CSV

  • Underspecified “family” of formats: different delimiters (,, ;, \t, |), quoting rules, header presence, encodings (UTF‑8 vs legacy), decimal separators, date formats, and null representations.
  • Real‑world files often violate RFC 4180: inconsistent quoting, wrong escaping, comments, misaligned rows, nested CSV in fields. A single robust parser for “all CSVs” is effectively impossible.
  • Locale and Excel issues: Excel changes separator based on locale, guesses types, converts IDs to numbers/dates/scientific notation, strips leading zeros, and can silently corrupt data on save. LibreOffice is often reported as more sane.
  • Encoding and schema are out‑of‑band; consumers must guess types and formats, leading to subtle bugs.

Debate over “human readable”

  • Pro: Human readability is seen as a major virtue. You can inspect, diff, and surgically fix data without code. CSV is considered durable and future‑proof.
  • Con: Human readability encourages ad‑hoc, buggy producers (“just join with commas”), making interoperability worse. Some argue binary + good tools beats text + misuse.

Parquet and other alternatives

  • Parquet praised for data pipelines: columnar, typed, schema‑aware, efficient for large datasets and partial reads; compresses well (comparable to csv.gz).
  • Criticisms: more complex spec, spotty or painful library support in some languages/environments, version quirks, not directly usable in Excel, and not ideal for row‑streaming use cases.
  • Other suggested formats: SQLite files, Arrow, Avro, Protobuf/Cap’n Proto, JSON/NDJSON, TSV, XML with schema, CSVY (CSV + YAML metadata), and “Unicode/ASCII separated values” using dedicated delimiter characters.

When CSV is appropriate vs not

  • Widely agreed:
    • Keep CSV (or XLSX) for external stakeholders and ad‑hoc analysis in spreadsheets.
    • Consider Parquet/SQLite/Arrow/etc. for internal, large, or performance‑sensitive pipelines where you control both ends.
  • Core tension: CSV is technically fragile but socially entrenched. Many see replacing it at scale as unrealistic; others argue we should at least stop using it as the default for “serious” system‑to‑system integration.