Water, Water, Everywhere
Let’s look at another account from the same telco—this time, experiencing water damage:
"The same year in California, another of our data centers had an incident with a fire protection sprinkler system that had lapsed out of its maintenance window. The system was triggered accidentally and soaked many of the libraries of backup tapes. Not only were the tapes not properly stored, but the offsite rotation schedule for those tapes had been ignored. As a result, many backups were ruined and lost."
There are a couple of lessons to be learned here. First, pay close attention to the maintenance schedule of equipment such as sprinklers and other fire-protection systems. Doing so will keep you out of trouble with the fire and code-enforcement departments, and prevent obvious issues like the one just described. Second, note the compound nature of the disaster. The sprinkler issue was a major disruption, to be sure, but look at how the disaster was magnified and compounded by failure to rotate backup tapes offsite.