Linux Crisis Tools

Pre-installing “crisis tools”

• Many argue for a standard “crisis-essential” package group, analogous to build-essential, installed on all servers ahead of time.
• Some already maintain such lists via config-management (Ansible/Salt/Chef) under names like devops_tools.
• There is concern that stripping systems and containers “for security” often removes vital debugging tools, making incidents much harder to resolve.

Cloud, cattle, and “just kill the machine”

• One camp praises virtualized “cattle” (often in cloud) where failing nodes can be quickly replaced, restoring service and debugging offline.
• Others push back: replacement can mask root causes, erode troubleshooting skills, and sometimes the issue persists on new instances.
• Several note that this “reboot/replace first” culture resembles junior devs recloning git repos instead of understanding problems.
• Some distinguish short‑term vs long‑term calculus: quick restarts are valid, but overused they lead to “technological mysticism.”

Storage, networking, and non-trivial failure modes

• Comments highlight that blindly replacing VMs is risky with complex storage (RAID, Ceph, MD, distributed filesystems); rebuilds can increase failure risk.
• Network and DNS issues are frequent and often misdiagnosed; there’s humor about “it’s never DNS… until it is.”
• On large “big iron” or HPC clusters, failures can involve orchestration layers (e.g., schedulers), tiered storage, and exotic interconnects.

Containers, Kubernetes, and serverless

• Tools remain useful on EC2 and within containers, since containers are just processes on a host; host-level tooling can still introspect them.
• For serverless (e.g., Lambda) with no OS access, people fall back to application-level observability/profiling.
• Security-focused Kubernetes setups often lack shells and tools; adding debug images or volumes of static binaries is suggested, but management may resist due to testing and scanning requirements.

Specific tools and approaches mentioned

• System and tracing tools: perf, eBPF tools, strace (as a “last resort”), lsof, fuser, sar family, atop (with caveats), tmux/screen.
• Network/disk/debug helpers: nmap, netstat, nc, tcpdump-in-Docker, ncdu, rsync, busybox, static rescue binaries.
• Some advocate Docker-delivered tools when package managers are broken, while others note this fails in air‑gapped environments.

Security vs operability

• One side worries that pre-installing powerful tools increases attack surface.
• Others counter that attackers can usually bring their own tools; the real cost is repeatedly being “locked in” almost tool‑less environments during crises.