Penalties and Lawsuits
Here's another way you can lose by not planning for disasters: contractual and non-performance penalties. One of the biggest companies in America, Dallas-based AT&T, is learning this lesson the hard way, according to an article in xchange Magazine. AT&T had a computer room disaster (caused by a water main break) that affected its wholesale customers, and AT&T's contracts with those customers contain performance measure (PM) provisions. If performance to wholesale customers degrades, AT&T owes those customers a lot of money under their contracts. In this case, that amount is potentially enough to get AT&T's legal staff involved. The AT&T lawyers are claiming that the contracts are not valid in this particular case. Their reason? God did it. This phenomena is known in the legal world as a force majeure event. AT&T is claiming force majeure in an attempt to dodge contractual penalties for the disaster. We submit that some disaster planning would have been cheaper. Read on.
A trade association called CompSouth (which represents AT&T competitors) is calling on state commissions to reject AT&T's force majeure claim, since if approved by regulators it would allow AT&T to avoid its contractually mandated performance-remedy payments to CLECs in nine states. Indeed, AT&T filed a notice that the disaster—a flood and subsequent power outage in its St. Louis data center in December—qualifies an act of God. Therefore, AT&T says, it doesn't owe CLEC customers for downtime they suffered as a result.
AT&T has a long history of trying to dodge performance payments to its competitors. (Take a look at our Dallas-based TelLAWCom Labs Inc. website for one example.) Setting this issue aside, however, the disaster AT&T is depending on to avoid penalties was preventable by any standard: On December 6, 2008, the basement of AT&T's St. Louis data center flooded because of a water main break. A power outage followed. So what's wrong with this picture? What's wrong is that serious contingency planners have been mediating concerns like water and power outages since Fred Flintstone bought his first stone PC. If you don't believe me, check out my 1993 book Writing Disaster Recovery Plans for Telecommunications Networks and LANS (Artech House Books). Virtually all of Chapter 5 discusses these very issues, complete with pictures and diagrams (see Figure 2) advising planners to install drains, place equipment away from sources of water, install uninterruptible power systems, and other standards and practices long since adopted by serious contingency planners. That's 1993, people—more than a decade and a half ago!
Figure 2 Reprinted with permission by Artech House Books.
By all indications, the AT&T issue will be settled in the courts or regulatory commissions. And AT&T's wholesale customers believe that the contract is in their favor. For example, AT&T didn't declare the event an act of God until February 10, 2009, despite obligations to do so right away. AT&T did tell its CLEC customers that the flood and outage had occurred, but waited two months to dub the incident an official force majeure in state commission filings. Perhaps that's how long it took for AT&T management to realize how much money was on the line?
"Not so fast," say AT&T's competitors. Under agreements between AT&T and CompSouth's 12 members, AT&T must promptly notify several entities of a force majeure event. AT&T is supposed to maintain certain operations support systems (OSS) performance metrics for its wholesale users—or pay up, unless the interruptions stem from an unforeseen act of God. To that end, AT&T said in February that the December flood was indeed a force majeure, and the company shouldn't have to compensate CLECs in nine states for the December 8–12 outages. CompSouth vehemently disagrees, and we vote with CompSouth.
A water main break is preventable, and standards for water damage in equipment areas have been established for 25 years or more. There is simply no excuse for something like this to happen. CompSouth argued in a February 19 filing with the Florida Public Service Commission that AT&T followed improper procedures in dealing with the flood and did not have backup redundancy in the data center, as it should have. This kind of issue is the contingency planner's worse nightmare.