How CERN serves 1EB of data via FUSE [video]

Original Article ↗ Hacker News Discussion ↗

CERN’s effort to serve an exabyte of scientific data over FUSE prompts scrutiny of how such massive storage is engineered and paid for, from commodity hardware running Ceph and the in‑house EOS filesystem to extensive tape archives and global replication tools like Rucio. Commenters contrast CERN’s relatively modest IT budget and heavy reliance on small, expert teams with the perception that “unlimited money” makes anything scalable, highlighting the role of open-source infrastructure and the impact of recent shifts toward more Microsoft-based solutions. The conversation also touches on why high‑energy physics facilities retain enormous historical datasets, emphasizing statistical significance in rare‑event measurements and the broader, often indirect societal benefits of fundamental research.

Storage Scale, Cost, and Architecture

  • CERN stores ~1 EB, mostly on Ceph and a homegrown distributed filesystem (EOS) over commodity hardware; commercial systems are mainly for tape.
  • A cited cost of ~1 CHF/TB/month (10+2 erasure coding) is debated:
    • Some see it as expensive at this scale and want a breakdown (hardware, staff, DC, networking, tape, etc.).
    • Others call it very cheap compared to universities charging >100× more, especially given tape, networking, and availability.
  • Bandwidth is noted as a major factor, but less so if data stays within a few data centers.

Data Ingestion, Reduction, and Backup

  • Experiments can generate ~1 PB/s of raw data; multi-stage trigger systems (including GPU-based software triggers) discard ~97% or more.
  • Backups rely heavily on tape and globally distributed replicas; not all tapes are themselves backed up due to cost and acceptable statistical loss.
  • Rucio is used to manage and replicate datasets across heterogeneous storage backends worldwide.

FUSE and Filesystem Concerns

  • FUSE is central to the approach, but performance concerns remain:
    • Context switching and metadata-heavy workloads are pain points.
    • Read-ahead and new features like FUSE passthrough help; io_uring integration is still work-in-progress.
  • Some users report issues with inotify over SSHFS/FUSE in containerized setups.

Budget, Resourcing, and Talent

  • CERN’s overall budget (1.4B EUR) and IT slice (50M EUR) are described as modest for the scale; rising energy costs even reduced accelerator run time.
  • Many argue “unlimited budget” is a myth; success comes from small, highly skilled, highly motivated teams and large in-kind contributions from member institutes.
  • Salaries and facilities are portrayed as unglamorous relative to Switzerland, but the scientific mission attracts top talent.

Open Source vs Microsoft Ecosystem

  • There was a major initiative to move away from Microsoft products toward open source; commenters say this later reversed under new leadership.
  • Some in the thread criticize a perceived “Microsoft push” (including partnerships/programs) and lament degraded user experience.
  • Others investigate leadership backgrounds and see this as part of broader institutional strategy rather than simple vendor capture.

Value of High-Energy Physics

  • A debate asks what practical social/economic benefits high-energy physics has produced.
  • Responses emphasize:
    • Basic research’s intrinsic value, not goal-driven utility.
    • “Side effects” such as advanced sensors, magnets, cryogenics, control and data systems.
    • Synchrotron light sources as a notable direct spin-off, heavily used in materials science and structural biology (e.g., early COVID-19 studies).

Reproducibility and Long-Term Data Retention

  • Experiments are, in principle, reproducible, but keeping historical data is crucial to achieve statistical significance against sensor noise.
  • Past “almost discoveries” that later resolved into noise underscore the need for long-term, large-scale datasets.

CERN as a Place and Culture

  • Commenters highlight the strong “mission” motivation versus typical profit-driven tech work and contrast it with adtech/banking.
  • CERN’s museum, tours, and on-site exhibits (including old accelerators and historical hardware) are praised as uniquely good at explaining cyberinfrastructure and big science.