Biggest shell programs

Original Article ↗ Hacker News Discussion ↗

Large, hand-written shell scripts—sometimes tens of thousands of lines long—are far more common and powerful than many developers realize, underpinning everything from CD rippers and firewalls to OS updaters, backup tools, and even order management systems. Contributors trade examples of massive Bash and POSIX shell programs, reflecting on why shell persists for complex automation: near-universal availability, strong OS integration, and ease of gluing tools together, despite poor readability, weak tooling, and portability headaches at scale. The conversation also highlights migration and evolution paths—Perl, AWK, PowerShell, and projects like Oil Shell—as ways to tame or replace sprawling shell codebases without losing the strengths of the Unix scripting model.

Examples of Large Shell Programs

  • Several large, single-file shell tools are cited: cd ripper abcde (5.5k LOC), ACME client acme.sh (7.1k), firewall tool FireHOL (20k), winetricks (22k), and steamtinkerlaunch (~24–27k).
  • OS-level tools: FreeBSD’s update client (3.6k) and poudriere (3× that) are mentioned as significant sh codebases; Arch’s makepkg is noted but relatively small (<5k).
  • Other sizable systems: backup/restore tool ReaR, rkhunter, a complex containerized pipeline manager (~6k), large test suites (e.g., ~10k lines of bash system tests for mod_pagespeed).

Portability, Environments, and Constraints

  • Shell is valued because it’s nearly always present, including on locked-down or air-gapped systems where installing compilers or new runtimes is hard or forbidden.
  • However, portability across shells and utilities is hard; behavior differences (e.g., DEC/Digital Unix truncating output at $COLUMNS) can dominate script size and complexity.
  • POSIX utilities like bc aren’t guaranteed on Linux, pushing some to rely on AWK or BusyBox as a predictable baseline.

Merits and Drawbacks of Large Shell Scripts

  • Pro: Shell (and Perl) excels at glue code, text processing, and messy OS interactions where Java/Python feel verbose. Some argue loss of Perl expertise has led to more manual drudgery.
  • Pro: Shell scripts are “transparent”: logic and checks are visible rather than hidden behind layers of data structures, making tools like rkhunter effective as human-readable documentation.
  • Con: Beyond a few hundred lines, bash is seen as hard to read, poorly tooled, and fragile (subtle scoping, error handling, quoting, and whitespace issues).
  • Con: Many advocate rewriting large scripts in C, Python, Perl, or other languages, or at least factoring out heavy logic into compiled or higher-level components.

Alternatives and Evolution (Perl, AWK, Oils, PowerShell)

  • Perl is repeatedly praised as the pragmatic migration path from large shell “abominations,” with stories of 50k-line shell systems replaced by ~5k lines of Perl and large performance gains.
  • AWK is used as a more pleasant, portable target language (including for interpreters of DSLs) when shell math and data handling become unwieldy.
  • The Oils project (OSH/YSH) is proposed as an evolutionary path from bash:
    • OSH focuses on high bash-compatibility and better diagnostics.
    • YSH adds strict error checking, real arrays/dicts, modules, and modern features while preserving shell-style pipelines.
    • Emphasis on gradual upgrade of existing bash scripts via shopt options.
  • On Windows, PowerShell is recommended over batch for new work, though legacy batch remains common.

Tooling, Debugging, and Safety

  • ShellCheck is widely recommended to catch common shell “footguns.”
  • Bash tracing (-x) and strict modes are used, but many still feel error handling is primitive.
  • For preventing accidental multi-line paste execution in terminals, bracketed paste mode via inputrc/bind is offered as a mitigation; enforcing rate limits or policies at the server level is seen as contentious because it may annoy users with different workflows.

Anecdotes and Attitudes

  • Numerous war stories describe:
    • Massive cross-Unix installers and CA systems mostly devoted to coping with utility differences.
    • Exotic behaviors on legacy Unixes and odd installers embedding archives/binaries inside shell scripts.
    • Personal “hacker spirit” scripts (VPN endpoint optimizer, weather-based roof sprinkler via X10).
  • Opinions diverge sharply:
    • Some love pushing shell to extremes and find it satisfying and fun.
    • Others see large shell programs as a sign that “life choices” should be reconsidered and favor more conventional programming languages.