Why pipes sometimes get "stuck": buffering

Original Article ↗ Hacker News Discussion ↗

User experiences and reactions

  • Many long-time Unix users admit they “know” about buffering issues but still get confused by stuck pipes, especially with commands like tail -f | grep | grep.
  • Several people report real-world surprises in CI pipelines, Docker logs, and other interactive debugging scenarios.
  • Some appreciate detailed, “dense” explanations of buffering; others worry about wordy, SEO-fluff articles and prefer summaries, sometimes via AI.

Buffering behavior and its tradeoffs

  • Commenters stress that buffering is essential: syscalls are expensive, screen and disk I/O are slow, and character-at-a-time writes are disastrous for performance.
  • Distinction is drawn between:
    • Unbuffered output (often unnecessarily slow and can interleave output incorrectly).
    • Line-buffered output (usually what users expect for pipes).
  • Multiple layers buffer data: stdio, TTY drivers, possibly the network; it’s “buffering all the way down.”

Proposed fixes and design debates

  • One camp argues for “threshold or timeout” flushing: flush after N bytes or after a short time. They note hardware analogies and interrupt/coalescing strategies.
  • Others push back: timers mean signals, threads, extra syscalls, and tricky interactions with POSIX, fork, error handling, and existing program expectations.
  • Some suggest event loops or flushing at the top of the main loop; others prefer explicit, predictable behavior even if user-hostile at times.
  • There is disagreement over whether this is a “real problem” vs. misuse of tools (interactive expectations on non-interactive contracts).

Tools and workarounds

  • Common workarounds mentioned:
    • grep --line-buffered, stdbuf (-oL -eL, or -i0 for input), and unbuffer.
    • Using a trivial cat or other simple stages to reshape pipelines, despite “useless cat” dogma.
    • Alternative single-process filters: grep -E 'thing1.*thing2', sed or awk combinations, PCRE lookaheads/\K.
  • Some note that not all awk/grep implementations buffer the same way, so behavior can be inconsistent.

Signals, pipes, and termination

  • Discussion on how SIGINT and SIGPIPE propagate in pipelines, and why Ctrl‑C often loses buffered data.
  • Clarification that line buffering defaults differ for terminals vs. pipes and that closing pipes early (e.g., grep -q) can break upstream commands expecting a consumer.