When you deleted /lib on Linux while still connected via SSH (2022)

Original Article ↗ Hacker News Discussion ↗

Directory hard links and Solaris specifics

  • Early subthread digs into whether you can hard-link directories to recover a moved /lib.
  • Consensus: POSIX forbids directory hard links, but some Solaris/illumos UFS implementations historically allowed root to do it, while ZFS and modern docs explicitly disallow it.
  • GNU ln even has a “try directories” flag for root, though it usually fails; underlying link() may or may not permit it depending on filesystem.

Static binaries, busybox, and modern recovery options

  • Older Unix/Linux systems often shipped statically linked tools in /sbin (or a static sln/busybox) to survive library loss.
  • Many distros have removed statically linked system binaries; recovery is now expected via initramfs, rescue media, or snapshots.
  • Several people describe recovering by:
    • Copying binaries from another identical machine (scp/rsync/uuencode+cat).
    • Reconstructing a static busybox or small helper program by echoing hex bytes over an existing executable path.
    • Using exec -a to satisfy busybox’s applet-name expectations.

Unlinked libraries and /proc tricks

  • Deleted libraries in use stay as “unlinked” (anonymous) inodes until last user exits.
  • In principle you can find inode numbers via /proc/$PID/maps and use filesystem tools (debugfs, zdb) to extract them.
  • Practically, this is hard because most tools you’d use are dynamically linked themselves; you may need to rely on shell builtins and then drop in a static tool over the network.

Human error stories and operational lessons

  • Many anecdotes: rm -rf /bin vs ./bin, rm -rf /, wiping /etc, removing all execute bits with chmod -R, even umount / on AIX.
  • Patterns of advice:
    • Don’t reboot immediately when things act weird; investigate first.
    • Avoid having humans type long destructive sequences; script and test them.
    • Keep two root shells when editing sudoers or critical SSH tunnels.
    • Pause and double-check before rm -rf or dd; some use echo, du -sh, or a # prefix as a safety step.

Safer deletion, trash vs snapshots, and tooling gaps

  • Suggestions: rm --preserve-root exists; people wish for a --preserve-home or top-level “are you sure?” prompts.
  • Opinions split between:
    • Using trash/recycle tooling (trash-cli, safe-rm) for day-to-day safety.
    • Relying instead on proper backups and/or filesystem snapshots (btrfs/ZFS), sometimes only for selected directories.
  • Some skepticism about trash tools’ reliability and snapshot bloat; others argue CoW snapshots are cheap and transformative.

Broader themes

  • Root’s lack of guardrails is highlighted as the root cause; immutable or restricted systems might reduce these incidents.
  • The situation is framed as a bootstrapping problem: as long as you have one running shell, one executable, and a way to overwrite it, you can theoretically rebuild a minimal system.
  • Techniques used here closely resemble “living off the land” methods from exploit writeups, but applied to self-rescue instead of offense.