False Smoke Alarm Nearly Floods Jane Street's NYC Datacenter

A systems engineer at Jane Street had to physically sprint through a skyscraper and manually yank power cables from every server in a machine room to prevent a false smoke alarm from triggering a catastrophic flood. The incident occurred because the building's safety protocols were designed to prioritize fire suppression over hardware preservation, creating a scenario where a dust-induced alarm nearly destroyed the firm's only New York datacenter.

The Mechanism of the Near-Flood

The datacenter's safety infrastructure followed a specific logic: if smoke is detected, the system shuts down the CRAC units to prevent oxygen from feeding a potential fire. Simultaneously, the smoke alarm signals the sprinkler system to fill its pipes with pressurized water.

Because the room was full of high-density server racks, the loss of cooling caused internal temperatures to rise rapidly. Since the sprinkler heads were heat-sensitive, the rising ambient temperature threatened to trigger a full-room flood even though no actual fire existed.

“And so my mission was made clear: as fast as I possibly could, get back to Jane Street, get into the machine room, and pull the power cord on every device in that room before the temperature climbed past 150 F and the sprinklers destroyed everything.”

— How a false smoke alarm almost drowned Jane Street’s only NYC datacenter

Manual Intervention and Risk Culture

Jane Street's risk culture prioritized 'big red button' manual overrides, a philosophy influenced by a partner with a nuclear submarine background. This culture dictated that the machine room be kept close to the trading floor so staff could physically intervene in emergencies.

When the alarm triggered, a systems engineer ran to the site and moved row by row, cabinet by cabinet, yanking power cables from every device. This manual effort successfully halted the temperature climb at 143.6°F, just below the point of sprinkler activation.

Telemetry and Future Safeguards

Following the incident, the team identified a weakness in their monitoring: building-level telemetry for cooling and sensors was not sufficiently reliable to catch 'unknown unknowns.'

To solve this, the team wrote a monitoring service in OCaml to query every Linux server in their fleet directly for onboard sensor data. By computing a simple mean of these temperatures, they created a more durable alert system that can detect generalized cooling failures regardless of whether building sensors are functioning correctly.