So you want to build your own data center

Original Article ↗ Hacker News Discussion ↗

A startup describing its move from hyperscale cloud to rented cages in colocation facilities prompted a deep look at what “building your own data center” really entails. Commenters probe the team’s hardware, networking (BGP, whitebox switches, PXE boot), and internal tooling choices, while weighing cost, egress fees, performance, and support trade-offs against the operational burden of managing physical infrastructure. Many argue the title overstates what colo provides compared to a true greenfield data center build, yet see this kind of metal ownership as a rational response to rising cloud costs and limited vendor support.

Scope of the project (colo vs “building a DC”)

  • Many point out the article is about cage colocation, not constructing a full facility.
  • Some see the title as misleading clickbait; others argue “your own data center” is commonly used for occupying/racking in a colo.
  • Multiple comments distinguish between: greenfield DC build (building + power + cooling), cage in an existing DC, and simple rack colo.

Motivations & economics

  • Strong interest in cost breakdown; Railway deferred detailed numbers to a future post.
  • Several argue cloud egress is “extortionate” and makes bare metal + colo vastly cheaper for bandwidth-heavy workloads.
  • Back-of-envelope comparisons: 100 Gbps at IXPs for a few thousand dollars vs millions in hyperscaler egress, though exact AWS/GCP pricing numbers are disputed.
  • Some warn that while hardware and transit can be cheap, you must factor in spares, repairs, on-call, and operational complexity.

Hardware, networking & design choices

  • Servers: Supermicro x86; ARM considered but current SKUs seen as too old for the scale/risk.
  • Networking: whitebox/SONiC + FRR, BGP in the data center, eBGP to hosts, BGP unnumbered; design inspired by “BGP in the Data Center.”
  • Current gen uses 25G leaf and 100G spine; commenters suggest alternative topologies (collapsed 100G per rack, CR3-class switches) for better $/Gbps.
  • PXE-based boot via pixiecore + Debian netboot; BMC/Redfish orchestration; custom host agent for QEMU VMs and BGP advertisement.

Tooling & DCIM

  • Railway built an internal DCIM/rack-modeling tool (“Railyard”) integrated with their orchestrator, rather than adopting Netbox/Nautobot.
  • Supporters say generic DCIMs force infrastructure to fit their data model; critics note Netbox’s complexity and performance issues.

Operations, reliability & process

  • Redundancy: RAID, dual feeds, spares on-site, remote hands; drive failures handled without staff travel.
  • Emphasis on standardized rack layouts, detailed cabling diagrams, and plans to use LLDP-based validation.
  • Some urge extensive fault-injection and gray-failure testing; discuss future live migration and user-coordinated maintenance windows.
  • Many share war stories about cooling failures, ad-hoc fans, animal-caused outages, and old-school DC chaos.

Cloud provider experiences & positioning

  • Railway cites poor support from a major cloud despite multi-million spend as a motivator.
  • Commenters contrast AWS, GCP, and Azure support: experiences range from outstanding (AWS/GCP) to frustrating (Azure and some AWS cases).
  • Debate over whether running your own metal is worth the long-term operational burden vs sticking with hyperscalers.

Alternatives & ecosystem

  • Mentions of dedicated bare metal (Hetzner, Hivelocity), OpenStack, Oxide racks, and colocation resellers as intermediate options.
  • Oxide is seen as technically attractive but too monolithic/early-stage for this use case.