Pack: A new container format for compressed files

Original Article ↗ Hacker News Discussion ↗

Overview of Pack

  • New archive/container format built on SQLite3 for structure and Zstandard for compression.
  • Aims for very fast handling of many files (especially many small files), random access to contents, simple CLI, and future OS integration.
  • Written in Pascal using FreePascal/Lazarus; designed so the code reads like pseudocode.

Performance and Benchmarks

  • Author claims large gains vs tar/zip/7‑Zip, especially on trees with tens of thousands of small files; cites examples like Linux source tree being packed in ~1–1.3s with smaller output.
  • Others rerun benchmarks and see smaller but real gains (often ~1.3–1.7× vs tar+zstd when both are well tuned), not the “orders of magnitude” implied by some graphs.
  • Some skepticism that part of the speedup is multithreading and Windows/NTFS pathologies rather than a fundamentally new idea.
  • Zstandard’s speed/ratio tradeoff is widely praised and seen as a big factor in Pack’s performance.

Use of SQLite3 and Zstandard

  • Supporters: SQLite file format is stable, well‑documented, widely used, and pledged compatible through 2050; makes alternative implementations easy if you use SQLite bindings.
  • Critics: Ties the spec to a single C implementation and its quirks; bad for streaming/pipes because SQLite wants a complete database; damaged‑file recovery and remote/NFS behavior may be tricky; SQLite not ideal as general interchange format.
  • Pack replaces SQLite’s magic bytes, making archives not directly openable in generic SQLite tools; there are custom tables (Content, Item, ItemContent) but no explicit on‑disk spec yet.
  • Author offers a CLI mode to transform Pack files to “real” SQLite3 for debugging and back, and uses a custom VFS with an internal versioning scheme.

Format Features and Design Choices

  • Random access and selective extraction (--include) are core goals; author reports extracting a single file from large archives in tens of milliseconds.
  • Files are chunked into ~8–32MB “contents”; incompressible regions (e.g., parts of media files) can be stored uncompressed, while other parts are compressed.
  • No filesystem metadata (owners, modes, timestamps) is stored; this is deliberate to avoid cross‑OS mismatch and leaking local details, though some users want optional metadata.
  • Currently no streaming/pipe support (tar | ssh tar–style); author says this is planned but not prioritized yet.
  • Compression tuning is intentionally simplified (normal vs --press=hard), which some users like, while others want finer control over Zstd levels.

Language Choice (Pascal)

  • Many are surprised to see a new systems tool in Pascal; reactions range from delight to “why not Rust/Go/C++?”.
  • Author argues Pascal is stable, readable, and long‑lived; some discuss Wirth‑family languages as emphasizing constraints and correctness vs the “C narrative.”

Comparisons to Existing Formats

  • Squashfs and dwarfs are mentioned as strong read‑only, random‑access compressed filesystems; Pack differs by being updatable (in principle) and more “user‑facing” as an archive tool.
  • sqlar (SQLite‑based archive) is cited as prior art; author shows Pack beats it significantly on speed and size in their tests.
  • Tar is criticized for being tape‑oriented, hard to parallelize, and lacking random access; others note its strength for streaming and pipes.
  • Zip is praised for ubiquity and simplicity but criticized for metadata quirks and poor documentation in edge cases.
  • ZPAQ/PAQ and high‑ratio compressors are mentioned; Pack explicitly targets speed + good enough compression, not extreme ratios.

Adoption, Portability, and Trust

  • Currently binaries exist for Windows and Linux; no macOS builds yet, and building from source requires following project‑specific scripts.
  • Some report runtime errors or size anomalies with early binaries; there are concerns about SQLite being compiled without thread safety while code appears multithreaded.
  • Several commenters are uneasy about anonymous maintainership and lack of a formal spec; author responds that trust should come from code, license, and reproducible builds, not identity.
  • General consensus: interesting idea with promising performance characteristics, but needs clearer specification, broader platform support, robust threading guarantees, and better documentation before it can be considered a serious “universal” replacement for tar/zip.