bzip3

Original Article ↗ Hacker News Discussion ↗

bzip3, a modern compressor inspired by bzip2’s Burrows–Wheeler transform, is drawing attention for sometimes beating zstd and other popular formats on compression ratio, especially for certain highly repetitive datasets. Commenters highlight that its advantages are highly data- and parameter-dependent, point out that some published benchmarks are misleading (e.g., mismatched window sizes and huge memory use), and note severe decompression overhead in some real-world tests. There is also debate over practical adoption—given zstd’s strong ecosystem support, licensing and naming choices for bzip3, and the fact that many users now prioritize fast, low-memory decompression over marginal gains in size.

Benchmarks and Fairness

  • Several commenters criticize the official benchmarks as rudimentary and “cherry‑picked.”
  • Main concerns:
    • bzip3 is tested with very large blocks (e.g., 512 MB) while zstd is left at default small windows (~8 MB), which severely handicaps zstd on repetitive corpora like concatenated source trees.
    • Memory usage is not normalized (e.g., cases with ~12–18 GB RAM for bzip3 vs <1 GB for zstd).
    • Some comparisons omit strong competitors (zstd in lrzip tests).
  • When zstd is re-run with a matching long window (--long), it can compress much smaller and faster than reported, sometimes beating bzip3 by large margins.

Real-World Performance and Data Dependence

  • Multiple independent tests show highly data-dependent behavior:
    • On some files, bzip3 beats zstd -19 in ratio at comparable or better speed.
    • On others, zstd wins slightly on size and decisively on speed.
  • For a Linux kernel tree, bzip3’s decompression was reported ~145× slower than multi-core zstd, with slightly worse compression.
  • On the enwik9 text corpus, bzip3 compressed substantially smaller than zstd but used orders of magnitude more RAM and time when decompressing.

Resource Usage and Practical Use Cases

  • A recurring theme: bzip3 can achieve impressive ratios, but decompression can be extremely slow and memory-intensive, especially with huge blocks.
  • Some suggest using it only for “compress once, decompress many times” scenarios after per-file algorithm tuning.
  • Others conclude it is “not terribly impressive” in aggregate benchmarks compared to modern alternatives.

Alternatives and Ecosystem

  • zstd is repeatedly described as the current “go-to” general-purpose compressor: very tunable, fast decompression, good ratios, wide support (filesystems, databases).
  • Benchmarks on JSONL logs show bzip3 giving the best ratio but with much higher CPU time; zstd is nearly as small as gzip but vastly faster than everything else at defaults, with room to tune.

Reliability, Licensing, and Naming

  • A strong “data may not be recoverable” warning alarms some, but others note identical disclaimers exist in bzip2, xz, and typical open-source licenses.
  • Some dislike reusing the “bzip3” name and the switch to LGPL instead of bzip2’s permissive license; others argue both naming and licensing are the author’s prerogative.

Future Directions

  • Ideas raised: multi-stream archives that auto-select algorithms per region, ML/agent-based parameter optimization, and formal verification of compressor correctness, though the latter is viewed as very difficult.