NTP at NIST Boulder Has Lost Power

Incident and Local Conditions

• NIST’s Boulder campus lost commercial power amid extreme winds (up to ~125 mph) and elevated wildfire risk; the site was officially closed with no onsite access.
• Power company preemptively shut off large areas to avoid a repeat of the Marshall Fire; this likely out-prioritized backup power at some facilities.
• WWV/WWVB broadcast time signals and other NIST sites (e.g., Gaithersburg, possibly Hawaii) remained largely operational.

Expected Impact on Timekeeping and Infrastructure

• Multiple commenters stress that “one site down” is a minor event: UTC is defined by an ensemble of ~hundreds of atomic clocks worldwide, not a single Boulder clock.
• Internet time is heavily redundant: pool.ntp.org, vendor pools (Google, Microsoft, Cloudflare, Ubuntu, Windows), GPS/GNSS-disciplined oscillators, and in-house atomic clocks at hyperscalers and telcos.
• Likely impacts are limited to: systems hardcoded to specific NIST Boulder servers; niche scientific setups or financial systems that explicitly require traceability to UTC(NIST) from that ensemble.
• Some note that financial regulations (e.g., CAT reporting) require traceability to NIST, but generally via local grandmasters and GPS; a short NIST distribution outage would more likely pause trading than corrupt it.

Clock Drift and Device Behavior

• Typical hardware clocks drift seconds per week; cheap microcontrollers with RC oscillators can drift seconds per day.
• Examples range from ThinkPads drifting a minute per month to consumer devices drifting ~1 minute per year.
• Datacenters rely on GPS + high-quality oscillators (OCXOs, rubidium/cesium standards) for holdover when GNSS is lost, aiming for sub‑microsecond accuracy over hours to days.

NTP Architecture, Redundancy, and Best Practices

• There are many Stratum 0 sources (atomic clocks); NIST Boulder is one contributor to a global ensemble.
• Best practice: configure multiple diverse NTP servers (≥4) so a single bad or missing source is outvoted. A dead server is safer than a wrong‑time server.
• Discussion covers NTP vs PTP, how stratum 0 clocks are re-synced, and rental calibration gear from NIST.

Debates, Skepticism, and Humor

• Some downplay any systemic risk; others worry about edge cases (databases, Kerberos, TLS, power grid synchrophasors), but mostly as thought experiments.
• There is pushback against exaggerated claims about clock-room sensitivity and facility design.
• Thread contains extensive time‑travel jokes, snark about banks and overdraft fees, and frustration over infrastructure funding, climate resilience, and utility undergrounding.