Single log line is 49KB+ (ext4) / 110KB+ (btrfs) of systemd-journald disk writes

Original Article ↗ Hacker News Discussion ↗

Linux administrators are reporting extreme write amplification from systemd‑journald, where a single log entry can trigger tens of kilobytes of disk writes on filesystems like ext4 and Btrfs, raising concerns about SSD wear and performance. Contributors attribute this to journald’s mmap-based, hash‑indexed binary log format and its mutation-heavy on-disk design, arguing it behaves more like a poorly designed database than a simple append-only log. Many suggest mitigating by limiting or offloading journald storage, or replacing it with traditional syslog or database-backed logging, and question why such a critical component was architected this way instead of using existing, battle-tested storage engines.

Disk write amplification & impact

  • A single log entry in systemd‑journald can trigger tens or hundreds of KB of disk writes, especially on btrfs, causing severe write amplification.
  • CoW filesystems (btrfs, others) magnify this, leading to large cumulative SSD writes on mostly idle desktops.
  • Many other desktop apps and services (KDE components, browser extensions, IPFS, Redis, etc.) also write frequently, compounding the issue.

Critiques of journald’s design and behavior

  • On‑disk format is seen as overcomplicated: scattered small writes, hash table updates, and mmap usage lead to extra IO and poor scaling.
  • Several commenters say reading with journalctl is slower than grepping plain text (even gzipped) logs.
  • Journald is reported as fragile (log corruption, past watchdog issues) and hard to debug or reason about.
  • Filtering and rate limiting are considered crude: hard to silence a single noisy source without impacting others; per‑service patterns exist but are limited and inconsistent.

Debate over mmap, databases, and formats

  • Strong criticism of using writable mmap for logs: little control over flush order, page‑granularity dirtying, and unpredictable behavior on complex filesystems.
  • Some argue mmap is not inherently wrong if combined with careful flushing; others see it as an unnecessary “clever trick”.
  • Multiple proposals to use existing storage engines instead of a custom format: SQLite, DuckDB, LevelDB, Parquet, or simple append‑only text plus offline indexing.
  • Disagreement over WAL/LSM approaches: some note WAL doubles writes; others argue it’s still more principled and efficient than journald’s current pattern.

Systemd ecosystem & governance concerns

  • Journald is frequently called the worst part of systemd; many would keep systemd but replace journald if it were actually optional.
  • Criticism that systemd is less modular than advertised because journald is effectively mandatory.
  • Complaints about project culture: defensive responses to bug reports, lack of open design review, tendency to reinvent rather than reuse proven components.

Workarounds and alternatives

  • Common mitigations: Storage=volatile, small size limits, or using journald only as a router while storing logs with rsyslog/syslog‑ng or external collectors.
  • Some users move to non‑systemd distros (Devuan, Void) or other OSes (e.g., FreeBSD) partly to avoid journald.
  • Suggestions to mark certain paths nocow on btrfs and to aggressively filter or blacklist noisy log messages where possible.