Why and how we’re migrating many of our servers from Linux to the BSDs

Original Article ↗ Hacker News Discussion ↗

Migrating servers from Linux to the BSD family is framed as a way to gain simplicity, predictability, and data integrity, especially when using tools like jails and ZFS instead of Docker and complex Linux storage stacks. Commenters compare real-world reliability of filesystems (btrfs, XFS, ZFS), debate Kubernetes and microservices versus “boring” but stable setups, and weigh BSD’s cleaner design and steady interfaces against Linux’s broader hardware support, ecosystem, and Kubernetes/container dominance. Many see BSDs as excellent for focused server and appliance use, but note that Linux remains the safer career and tooling bet, particularly for container-heavy environments.

Reaction to the migration story

  • Several commenters appreciate seeing concrete results from Linux→BSD migrations but want more technical depth (e.g., what exactly caused XFS failures, why workloads sped up).
  • Some consider running three different BSDs as “not solving problems” due to operational overhead; others see tailoring OS to task as a positive.

Filesystems: XFS, ext4, ZFS, Btrfs, UFS

  • XFS: Mixed history. Older Linux XFS on cheap hardware was described as fragile under power loss; modern XFS is widely praised as fast and reliable, especially with mdadm. Some want more detail on the author’s specific XFS incident.
  • ext4: Seen as “boring but safe,” with strong recovery tools and design tuned for commodity PCs.
  • ZFS: Many like ZFS (especially on FreeBSD, Solaris, and appliances like TrueNAS); others complain about Linux ZFS kernel-compatibility friction and higher resource use. One notes serious ZFS bugs and a contributor regretting the encryption merge.
  • Btrfs: Deeply polarizing. Some report years of trouble-free use (including large-scale deployments), others repeated catastrophic failures, especially when volumes fill or snapshots hide space usage. RAID5/6 is widely regarded as unsafe.
  • UFS/soft updates: Praised for simplicity, metadata safety, snapshots, and background fsck; seen as slower but robust.
  • General theme: real-world experience varies wildly; hardware quality, RAID level, and usage patterns matter.

BSD vs Linux: strengths and weaknesses

  • BSD positives: cohesive “full OS” design, stable interfaces, simple init (rc.conf), jails, strong docs and manpages, predictable upgrades, and good ZFS integration. Often preferred for NAS, routers, firewalls, and small servers.
  • BSD negatives: weaker desktop, Wi‑Fi, GPU and power‑management support; historically weaker Java support; some rough edges in ZFS booting and /dev semantics; smaller ecosystem and job market.
  • Linux positives: broader hardware and software support, containers/Kubernetes ecosystem, and more people familiar with it.
  • Some recommend FreeBSD (or TrueNAS) specifically for ZFS file servers.

Containers, Kubernetes, and orchestration

  • Lack of a mature Kubernetes equivalent on BSD is a blocker for teams heavily invested in microservices.
  • Opinions on Kubernetes are split:
    • Pro: common platform across enterprises, consistent deployment model, cloud-managed offerings hide much complexity.
    • Skeptical: seen as “ornamental/accidental complexity,” expensive, overkill for non–hyper-growth businesses, and often adopted as cargo cult.
  • Alternatives mentioned: jails plus tools like BastilleBSD, systemd-nspawn, microVMs; some argue traditional VMs plus config management can be simpler.

Operational practices and client trust

  • Major debate over the author migrating client VMs to BSD “without telling them.”
    • One side: the OS is an implementation detail; if SLAs are met and contracts allow infrastructure changes, silent migration is fine.
    • Other side: changing OS carries high risk; clients should be warned of potentially disruptive work even if it ends up improving performance.
  • Some stress the need for maintenance windows and clear communication; others say providers routinely change internals (like cloud vendors) and customers don’t care so long as service quality remains.

Careers, economics, and meaning of IT work

  • A thread explores how IT “optimization” mainly enriches owners while automating away workers’ jobs, feeding a sense of meaninglessness.
  • Mid-career workers describe feeling trapped in “soulless” but well-paid roles, unable to retrain without risking family stability.
  • Others counter that modern living standards and housing expectations complicate claims of generational poverty, but acknowledge decoupling of meaningful work from livelihood.