An Alarm Nobody Got: The Microsoft Outage, a Hotel Pool, and the Fail-Safe That Wasn’t

Friday afternoon, November 13, around 4 p.m. local time. The fire service in Brühl, Germany, gets called to Hotel Ling Bao, the themed hotel at the Phantasialand amusement park. Guests who’d been in the pool were complaining of health problems.

By the end, the rescue service had triaged 15 people. Fourteen were classified as lightly injured and one as moderately injured. Four went to the hospital, and one stayed for observation.

Then came the twist. The next day, the park told German broadcaster WDR that the pool’s safety system had already sent an alarm. By email. And because of the Microsoft outage, those emails never arrived.

Mira, you’ve probably seen the viral headline making it sound like Microsoft hurt those guests. That’s not what happened. The real story is less dramatic and way more useful for anyone who runs a control system.

What actually happened

Two separate failures lined up. Let’s take them one at a time.

Failure one: the chemistry. Fire service tests at the hotel found a defect in the automatic dosing system, which led to a clearly elevated chlorine level in the pool water, as WDR reported. Matthias Stupp of the Brühl fire service told the Kölner Stadt-Anzeiger the defect caused an overdose of chlorine in the pool. The water was replaced, and the pool reopened. On Friday evening, Stupp said how the defect happened was still unclear.

WDR says 13 guests first complained of respiratory irritation before the count rose to 15. Some outlets, like Radio Bonn and WDR’s own video titles, describe chlorine gas released in the pool area. The fire official’s quoted wording is an overdose of chlorine in the pool. I’m keeping both and attributing each one.

Failure two: the alarm. According to Phantasialand, speaking to WDR on Saturday, the sauna area’s maintenance safety system had already raised an alarm by email. Because of the Microsoft disruption, no emails came in, so nobody noticed.

That link comes from the park. Officials didn’t make it, and Microsoft hasn’t commented on the hotel at all. Nobody has said the dosing controller itself ran on Microsoft 365. The outage didn’t put chlorine in the water. It just, per the park, kept the warning from reaching people.

The outage was real. Microsoft confirmed a global Microsoft 365 and Exchange Online outage starting Thursday, November 12. Its status account was clear: “This was a technical issue not related to cyber or AI / agents.” Microsoft executive Charles Lamanna posted a preliminary account describing a latent bug, extra load and missing capacity that together caused a cascading failure, and he wrote that “at its lowest, more than 80% of servers were unresponsive.”

So no hacker, amigos. No cyberattack angle here.

One detail to know: local reports put about 30 Brühl firefighters on scene, while Cybernews reported about 40 first responders. Different counts, different scopes, so I’m not merging them. And one local radio report said Thursday afternoon, but the fire official’s account and WDR both point to Friday.

Why this matters for OT

This part is my analysis, not an official finding against the hotel. We don’t know what local alarms existed, failed, or were never installed.

Here’s the thing. A notification is not a safety function. If the only path between “the chemistry is wrong” and “a human does something” is a cloud inbox, then your safety depends on a service you don’t control.

Think of it like a smoke detector that doesn’t beep. It just sends you a text. Works great, until your phone is dead. Then the kitchen is on fire and the hallway is quiet.

WDR quoted cybersecurity architect Stephan Skrobisch, who put it bluntly: the problem is the system setup, because nobody implemented a second warning mechanism. WDR’s example was an acoustic signal. An operator at another spa told WDR his technician measures water values three times a day and that a red lamp lights up if a pool is over or under chlorinated. A red lamp doesn’t care if Exchange is down.

NIST SP 800-82r3 says it straight: “OT systems typically employ fail-to-a-known-state design (i.e., fail-safe design).” And it adds that “the loss of network communications, could trigger these fail-safe events.” Losing the email path is a loss of communications. A good design treats that as a reason to go safe, not a reason to go silent.

And the chemical feed itself? CDC’s Model Aquatic Health Code, a U.S. model code and not German law, requires “an automatic means to disable all chemical feeders” on low or no flow. For new equipment, it requires that a disabled feeder trigger “a visual alarm or other indication” so staff on site can evacuate bathers. I’m not saying that’s what failed in Brühl. I’m saying that’s what layered protection looks like on paper.

What to do about it

Familia, go look at your own plant, pump station, or building with this lens:

  1. Map every alarm path end to end. For each safety-relevant alarm, write down how it actually reaches a human. If the answer is “email” or “a cloud app,” that’s a dependency, not a design.
  2. Add an out-of-band path. Local horns, beacons, HMI alarms, and a phone or SMS path that doesn’t share the same provider. Two paths that fail together are one path.
  3. Fail to a known state. If a feed goes out of range, shut the feed down locally. Don’t wait for someone to read a message.
  4. Interlock the chemistry. Chemical feeders should stop on no flow or out-of-range readings, and the interlock should live in the field, not in the cloud.
  5. Monitor the monitor. If no alarm or heartbeat message arrives on schedule, that silence should itself raise an alarm.
  6. Keep manual checks. Scheduled hands-on readings, like that three-times-a-day routine, catch what automation misses.
  7. Test during outages. Next time your email or cloud provider has a bad day, ask what alarms would have gone nowhere. Better yet, drill it before it happens.

Bottom line

  • The dosing defect caused the overdose. The outage, per the park, only hid the warning, and that’s exactly why alarm design matters.
  • A safety alarm that only lives in an inbox is not a fail-safe.
  • Build local, out-of-band, fail-to-safe protection so a cloud outage stays an IT problem.

Stay sharp out there.

Javier

Sources

  1. https://www1.wdr.de/nrw/rheinland/rhein-erft-kreis/technischer-defekt-chlor-phantasialand-1-100.html
  2. https://www.ksta.de/region/rhein-erft/bruehl/phantasialand-bruehl-mehrere-verletzte-nach-chlor-alarm-im-hotel-1416501
  3. https://www.radiobonn.de/artikel/chlorgas-im-phantasialand-hotelgaeste-verletzt-2806660
  4. https://x.com/MSFT365Status/status/2121303408313303423
  5. https://x.com/clamanna/status/2122209697226948729
  6. https://cybernews.com/security/microsoft-outage-chlorine-poisoning/
  7. https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-82r3.pdf
  8. https://web.archive.org/web/20241222173945/https://www.cdc.gov/model-aquatic-health-code/media/pdfs/Mini-MAHC-CODE-Chlorine-Gassing-508.pdf

Leave a Reply