High-speed 10Gbps full-mesh network based on USB4 for just $47.98
A blog post showing three USB4‑equipped mini PCs wired together as a cheap 10 Gbps “full‑mesh” alternative to rack servers prompted debate over both networking and power economics. Commenters compare Thunderbolt/USB4 peer‑to‑peer links with used 10–100 Gbps Ethernet and Infiniband gear, noting trade‑offs in latency, scalability, cable length, and offload features, while others challenge the author’s assumptions about California electricity prices and 1 kW server power draw as wildly overstated. Overall, many find the USB4 approach clever for small, quiet home labs, but see traditional server hardware and SFP+‑based networks as more robust and often similarly priced once the full system cost is considered.
USB4 / Thunderbolt networking concept
- Many find the USB4/TB direct-link cluster a clever reuse of ports that “already exist” on modern mini‑PCs, akin to FireWire or old null‑modem/parallel‑cable tricks.
- Others note it’s inherently short‑range (a few meters) and cable costs rise quickly for high‑speed certified cables, so it’s not suited to whole‑house wiring.
Performance and technical limits
- Reported ~11 Gbps throughput is seen as underwhelming relative to USB4/TB headline speeds.
- Suggested causes:
- Single TCP stream limits; multiple iPerf streams and jumbo frames might raise throughput.
- Lack of NIC‑style offloads shifts packet processing to CPU, hurting single‑stream throughput and latency.
- Possible bandwidth sharing between both USB4 ports on a single controller.
- Some consumer “USB4” ports are only 10 Gbps.
- There is disagreement whether the spec or Intel limits TB Ethernet emulation to 10 Gbps; counterexamples (up to ~16 Gbps on Macs) are cited. Overall cap is unclear.
Topology and scaling
- Network interfaces are configured as point‑to‑point /32 links, matching the physical full‑mesh of three nodes.
- Commenters note:
- This is a fully connected topology, but with 3 nodes it’s not very “mesh‑like” or a ring; it doesn’t route traffic through intermediates.
- Hardware and cabling scale quadratically with node count, making it impractical beyond a handful of machines.
- All forwarding happens on CPUs, which doesn’t matter much for small K8s/virtualization setups but wouldn’t scale.
Comparison to traditional Ethernet / DC gear
- Some argue 10 GbE/25 GbE/40 GbE via used Mellanox/Intel SFP+/QSFP NICs and DAC/fiber is cheap and better understood; examples show ~$20–30 NICs and low‑cost optics, plus inexpensive 8x10G Mikrotik switches.
- USB4 networking latency is reported worse than switched Ethernet, and bandwidth is viewed as more “unpredictable” than classic Ethernet links.
Power, cost, and hardware trade‑offs
- Several posts challenge the article’s power math:
- Rack servers with 750–1000 W PSUs rarely draw that continuously; real‑world R6xx/R7xx numbers are ~100–150 W idle and ~200–350 W under load.
- Electricity‑rate data cited in the article is called outdated; Bay Area and PG&E/SDG&E users report residential rates 2–3× higher than 15¢/kWh, sometimes approaching $1/kWh all‑in.
- Some call the “$50 mesh” framing misleading because it assumes already‑owned, relatively pricey USB4 mini‑PCs; eBay 10G/25G NICs plus a used switch can be comparable in total cost.
- Mini‑PC storage expandability (limited bays, expensive NVMe vs spinning disks) is seen as a practical downside vs rack servers or dedicated NAS.