NIST was 5 μs off UTC after last week's power cut
Trust in NIST, Scope of the Incident, and Redundancy
- Several commenters argue NIST’s transparency and handling of the outage increase trust rather than reduce it.
- Others note the headline “NIST was off UTC” is misleading: only the Boulder servers were affected; other NIST sites stayed correct.
- Properly designed systems should not depend on a single time source; using ≥4 independent NTP sources plus GPS is repeatedly recommended.
How Bad Is 5 µs?
- Multiple people stress that 5 microseconds is negligible for Internet NTP users, where network jitter is typically ~1 ms.
- Concern during the outage was the unknown state immediately after power restoration: bad time could cause large step changes if clients trusted it, so “no time” is safer than “unknown time.”
- Once the offset is known and bounded, a small, decaying 5 µs error is considered operationally harmless for almost all users.
Time Sources and Architectures
- High-precision users typically rely on:
- GPS / GNSS with local oscillators (OCXO, rubidium, cesium, hydrogen masers) for holdover.
- Precision Time Protocol (PTP) and variants like White Rabbit over dedicated networks or dark fiber.
- NIST’s “Time Over Fiber” service for ultra-precise, GPS-independent distribution.
- NTP over the public Internet is seen as a coarse layer; serious applications use local stratum-1 servers and hardware references.
NTP Pool and Security Concerns
- Some warn that NTP pool servers can be used as IPv6 reconnaissance “honeypots” and that you don’t control which servers you hit.
- Others report poor reliability from pool.ntp.org in large deployments and prefer major vendors’ time services (Google/Microsoft/Apple).
Who Actually Needs Micro/Nanosecond Accuracy?
- Cited use cases include: high-frequency and low-latency trading, 4G/5G telecom, radio/particle physics experiments, spacecraft state vectors, GPS itself, distributed radio telescopes, lightning detection, robotics sensor fusion, audio/video and simulcast radio synchronization, and globally distributed databases (e.g., Spanner-like systems).
Synchronization Techniques and Software
- Discussion highlights GPSDOs, rubidium/CSAC references, PTP/White Rabbit, and careful timestamping pipelines.
- chrony is praised as more robust than many OS-default NTP clients, and some environments disable continuous NTP to avoid clock jumps when PTP is also disciplining the clock.
Meta: Titles and Impact
- Several commenters describe the phrase “microseconds from disaster” as clickbait, given the tiny offset and extensive redundancy.
- Nonetheless, a few note that even small timing anomalies can have financial or analytical implications at the margins.