A false smoke alarm almost drowned Jane Street's only NYC datacenter
A near-disaster at Jane Street’s only New York City datacenter, where a false smoke alarm shut off cooling and nearly triggered ceiling sprinklers, prompts scrutiny of how financial firms design and protect on-premise infrastructure. Commenters contrast water-based fire suppression with gas and “clean agent” systems, and recount past incidents like Hurricane Sandy, when staff hauled diesel by hand to keep generators running. Many see these “heroic” saves as symptoms of poor disaster planning, weak redundancy, and workplace norms that quietly exploit employees’ proximity and willingness to go above and beyond.
Work–life boundaries and on‑call expectations
- Several commenters recount being informally designated “emergency” or on‑call staff simply because they lived close to the office or had unusual sleep habits.
- Common lesson: oversharing personal details (insomnia, being single, lack of weekend plans, living nearby) often leads to unpaid or under‑compensated extra work.
- Multiple people emphasize the importance of saying “no,” drawing clear responsibility boundaries, and demanding explicit on‑call pay or raises if 24/7 availability is expected.
- Others argue that becoming socially “vapid” at work is too high a personal cost, preferring normal collegial sharing plus firm boundaries.
Fire suppression and cooling in data centers
- Many are surprised a trading firm used sprinklers at all; “clean agent” gas systems (historically Halon, now substitutes or CO₂ / inert gases) are seen as more appropriate for server rooms.
- Commenters note safety and code constraints: gas systems can endanger people, especially in mixed‑use office buildings, and older Halon is now banned, so newer gases or low‑oxygen designs are used.
- Gas discharge can damage spinning disks via sound pressure; low‑oxygen and inert‑gas systems introduce their own operational and human‑safety complexities.
- Several question why Jane Street’s CRAC units were wired to shut off on smoke alarm; others say their designs intentionally keep CRACs running to avoid exactly this overheating scenario, relying on code allowances.
Hurricane Sandy and real‑world DR failures
- Thread is rich with first‑person Sandy stories: flooded New York data centers, diesel being moved by hand (55‑gallon drums vs. more realistic 5‑gallon buckets), and ad‑hoc “bucket brigades” of staff and volunteers.
- Generators placed on higher floors avoided flooding but relied on basement fuel systems or pumps that failed when submerged.
- Fuel quality (old, contaminated diesel) caused rapid clogging of generator fuel filters; people describe learning emergency maintenance on the fly.
- There are anecdotes of servers and storage physically carried out of buildings, improvised backup strategies, and painful lessons about provider couplings and missing fittings.
Risk management, backups, and ‘hero culture’
- Strong debate over whether these dramatic “save the data center” efforts should be celebrated.
- One side: heroism of individuals under bad conditions is admirable, even if management failed.
- Other side: these events mostly expose poor planning; stories that glorify the rescue can normalize under‑investment in redundancy and backups.
- Some argue leadership often prefers cheap heroics and “pizza and praise” over proper redundancy and off‑site backups; others note that maintaining redundant sites is expensive, and very rare events may be handled “manually” for cost reasons.
- Several point out that preventive work and boring competence are undervalued compared with flashy firefighting; comparison is made to Y2K and vaccines as “invisible” successes.
Datacenter siting and design tradeoffs
- Some are surprised a latency‑sensitive trading firm ran a data center in Manhattan at all, given most exchanges operate out of New Jersey facilities.
- Others note this was an earlier era when many firms simply had “server rooms” in offices, and proximity for hands‑on work was important.
- Commenters highlight that in skyscraper office contexts, sprinklers are mandated for life safety; letting a data‑center fire burn is not an option.
Miscellaneous technical and cultural notes
- Discussion touches on EPO (emergency power‑off) expectations; several are surprised there wasn’t a single shunt‑trip or EPO button rather than yanking power cords.
- Some mention designing scripts to monitor temperature (or even over‑cooling) via server sensors to catch HVAC failures early.
- Side threads cover wabi‑sabi (imperfection as aesthetic), OCaml jokes about alarm firmware, and nostalgia for older telephone and VAX machine rooms.