Ask HN: How should organize and back up 23 TiB of personal files?

Hacker News Discussion ↗

Managing 23 TiB of personal data raises hard tradeoffs between simplicity, cost, and long‑term reliability. Commenters converge on combining large local storage (often NAS or DIY servers with ZFS/Btrfs, RAID, and periodic scrubs) with at least one off‑site copy, using tools like Borg, restic, or rclone for encrypted backups to services such as Backblaze B2, S3 Glacier, or cheap VPS storage. Many emphasize that pruning unneeded files, separating truly critical data from bulk media, and regularly testing restores matter more than any specific technology choice, and that there is no single “ideal” setup—only what best fits an individual’s budget, risk tolerance, and effort.

Scope & priorities

  • 23 TiB is large enough that organization and curation matter as much as technology.
  • Several comments stress first classifying data by importance (disastrous / upsetting / disappointing / meh to lose) and deleting non‑essential or duplicate content.
  • Tools suggested for cleanup/dupes: WinDirStat / WizTree, fdupes, fclones, czkawka.

Local storage strategies

  • Many favor a home NAS or generic x86 box with multiple HDDs.
  • Common filesystems: ZFS (raidz or mirrors, scrubs, checksums, optional encryption), btrfs (snapshots, compression, dedup with bees), ext4 for simplicity.
  • Mixed views on RAID: useful for uptime and some redundancy, but repeatedly noted “RAID is not a backup.”
  • Recommendations range from simple mirrored disks (2–3× 18–24 TB) to 4+ disk arrays (RAID5/6, ZFS raidz) with regular scrubs and integrity checks.
  • Some suggest Synology/QNAP/UnRAID for ease of use and integrated tools; others prefer DIY for flexibility and cost.

Backup topology & off‑site copies

  • Strong support for 3‑2‑1‑style setups:
    • Primary storage.
    • Local backup (another disk/NAS; rsync/borg/restic, snapshots).
    • Off‑site copy (cloud or physically at relatives/safe deposit box).
  • Low‑tech options: periodically clone to new HDDs, store in attic/parents’ house, swap drives every few months, test restores.

Cloud and remote backup tools

  • Software: borg, restic, rclone (with client‑side encryption), rsync, Arq, Git Annex, BorgBase, HashBackup.
  • Cloud targets mentioned: Backblaze (personal and B2), S3 Glacier/Deep Archive, Google Archive storage, Wasabi, InterServer VPS, rsync.net.
  • Repeated emphasis on client‑side encryption and “untrusted server” models.
  • Concerns about:
    • Costs at tens of TB (Tarsnap widely criticized as prohibitively expensive here).
    • Egress/deletion penalties (Glacier, some cold storage).
    • Privacy and server‑side decryption (Backblaze filename leakage; restore‑time decryption on server).

Alternative media & niche ideas

  • LTO tape (especially used LTO‑6) suggested as cost‑effective long‑term archival once hardware is acquired; requires some setup and planning.
  • Blurays mentioned for dead media.
  • A few propose using the project itself as a learning/portfolio exercise (e.g., custom S3‑based system).

Meta‑point

  • No single “right” solution emerges; optimal setup depends on budget, access needs, threat model, and willingness to maintain tooling.