Ask HN: How should organize and back up 23 TiB of personal files?

Hacker News Discussion ↗

Scope & priorities

  • 23 TiB is large enough that organization and curation matter as much as technology.
  • Several comments stress first classifying data by importance (disastrous / upsetting / disappointing / meh to lose) and deleting non‑essential or duplicate content.
  • Tools suggested for cleanup/dupes: WinDirStat / WizTree, fdupes, fclones, czkawka.

Local storage strategies

  • Many favor a home NAS or generic x86 box with multiple HDDs.
  • Common filesystems: ZFS (raidz or mirrors, scrubs, checksums, optional encryption), btrfs (snapshots, compression, dedup with bees), ext4 for simplicity.
  • Mixed views on RAID: useful for uptime and some redundancy, but repeatedly noted “RAID is not a backup.”
  • Recommendations range from simple mirrored disks (2–3× 18–24 TB) to 4+ disk arrays (RAID5/6, ZFS raidz) with regular scrubs and integrity checks.
  • Some suggest Synology/QNAP/UnRAID for ease of use and integrated tools; others prefer DIY for flexibility and cost.

Backup topology & off‑site copies

  • Strong support for 3‑2‑1‑style setups:
    • Primary storage.
    • Local backup (another disk/NAS; rsync/borg/restic, snapshots).
    • Off‑site copy (cloud or physically at relatives/safe deposit box).
  • Low‑tech options: periodically clone to new HDDs, store in attic/parents’ house, swap drives every few months, test restores.

Cloud and remote backup tools

  • Software: borg, restic, rclone (with client‑side encryption), rsync, Arq, Git Annex, BorgBase, HashBackup.
  • Cloud targets mentioned: Backblaze (personal and B2), S3 Glacier/Deep Archive, Google Archive storage, Wasabi, InterServer VPS, rsync.net.
  • Repeated emphasis on client‑side encryption and “untrusted server” models.
  • Concerns about:
    • Costs at tens of TB (Tarsnap widely criticized as prohibitively expensive here).
    • Egress/deletion penalties (Glacier, some cold storage).
    • Privacy and server‑side decryption (Backblaze filename leakage; restore‑time decryption on server).

Alternative media & niche ideas

  • LTO tape (especially used LTO‑6) suggested as cost‑effective long‑term archival once hardware is acquired; requires some setup and planning.
  • Blurays mentioned for dead media.
  • A few propose using the project itself as a learning/portfolio exercise (e.g., custom S3‑based system).

Meta‑point

  • No single “right” solution emerges; optimal setup depends on budget, access needs, threat model, and willingness to maintain tooling.