Linux Crisis Tools
Engineers trade experiences on what tools and practices are essential for diagnosing and recovering from severe Linux production incidents, from classic utilities like ssh, nmap, strace and sysstat to newer eBPF-based profilers. A recurring tension is whether to quickly replace misbehaving cloud instances and containers versus investing time in deeper root-cause analysis, especially when modern “security-first” images strip out shells and debuggers. Many argue that pre-installing rich observability tools on servers and providing clear runbooks pays off in faster, safer crisis handling, despite concerns about attack surface and operational overhead.
Pre-installing “crisis tools”
- Many argue for a standard “crisis-essential” package group, analogous to build-essential, installed on all servers ahead of time.
- Some already maintain such lists via config-management (Ansible/Salt/Chef) under names like devops_tools.
- There is concern that stripping systems and containers “for security” often removes vital debugging tools, making incidents much harder to resolve.
Cloud, cattle, and “just kill the machine”
- One camp praises virtualized “cattle” (often in cloud) where failing nodes can be quickly replaced, restoring service and debugging offline.
- Others push back: replacement can mask root causes, erode troubleshooting skills, and sometimes the issue persists on new instances.
- Several note that this “reboot/replace first” culture resembles junior devs recloning git repos instead of understanding problems.
- Some distinguish short‑term vs long‑term calculus: quick restarts are valid, but overused they lead to “technological mysticism.”
Storage, networking, and non-trivial failure modes
- Comments highlight that blindly replacing VMs is risky with complex storage (RAID, Ceph, MD, distributed filesystems); rebuilds can increase failure risk.
- Network and DNS issues are frequent and often misdiagnosed; there’s humor about “it’s never DNS… until it is.”
- On large “big iron” or HPC clusters, failures can involve orchestration layers (e.g., schedulers), tiered storage, and exotic interconnects.
Containers, Kubernetes, and serverless
- Tools remain useful on EC2 and within containers, since containers are just processes on a host; host-level tooling can still introspect them.
- For serverless (e.g., Lambda) with no OS access, people fall back to application-level observability/profiling.
- Security-focused Kubernetes setups often lack shells and tools; adding debug images or volumes of static binaries is suggested, but management may resist due to testing and scanning requirements.
Specific tools and approaches mentioned
- System and tracing tools: perf, eBPF tools, strace (as a “last resort”), lsof, fuser, sar family, atop (with caveats), tmux/screen.
- Network/disk/debug helpers: nmap, netstat, nc, tcpdump-in-Docker, ncdu, rsync, busybox, static rescue binaries.
- Some advocate Docker-delivered tools when package managers are broken, while others note this fails in air‑gapped environments.
Security vs operability
- One side worries that pre-installing powerful tools increases attack surface.
- Others counter that attackers can usually bring their own tools; the real cost is repeatedly being “locked in” almost tool‑less environments during crises.