Of Toilets and Technology
There’s only one kind of disaster: the messy kind. If you’ve ever had water in a place in your building where it didn’t belong, you understand what I mean by messy. Another example that comes to mind for me is a client whose cable risers had metal mesh floors that opened from one floor up and down to adjoining floors. These risers also contained water pipes—just like yours probably do. To add to this issue, the cable risers were adjacent to the men’s and women’s restrooms. We cited this risk in a vulnerability analysis that we performed for that client. We wrote the client up not so much because of the pipes but because if one of the restroom toilets backed up, the water only really had one place to go—straight down the riser. Note to my readers: My wife and I know from firsthand experience that every kid backs up a toilet a minimum of three times during childhood. With our seven children, 21 episodes of backed-up toilets lent us a high degree of sensitivity to this issue. All kidding aside, we determined that water could be a problem for this client, particularly given the proximity to the risers and associated electronic equipment.
About three months after our report, we received a call one Monday morning that the client had sustained a disaster. It was water in the risers, all right, but not from the restrooms. Apparently one of the pipes inside the riser had sprung a leak early on Sunday. The water fell four stories to the bottom of the riser in the subbasement, to the only place in the whole organization that it absolutely should not go—the main UPS (uninterruptible power system). It was conveyed to me that the flash could be seen from the subbasement all the way to the third floor, and all primary and backup power instantly went "poof." Imagine the impact of a hard shutdown on all of your mainframe, open, and telecommunications systems at this moment. Now multiply it by about 10, because this particular client was huge. Much to their credit, business was restored by end of day on Monday, but things were touch-and-go all day, and it was several more days before their organization felt "right" again.
There are a few lessons to be learned by this example. One is about water. Another is about UPS and the false sense of security you may have, thinking that UPS systems don’t fail. They do. This company eventually mitigated the future risk by mandating that outdoor electrical fixtures (that are waterproof) would become the standard for all future installations. Because of their disaster experience, this company created a new standard.