IT Management Reference Guide

The previous section on Lessons Learned from Real-Life Disasters discussed five suggestions on how to prepare a business environment, particularly the IT department, for a major disaster. Those lessons applied to any event, whether it be an act of nature or manmade. In this section I describe five additional lessons that were learned during and immediately after the occurrence of various disasters.

1. Establish and utilize emergency contracts with key suppliers. A few years ago a California defense contractor client of mine experienced a major earthquake. Among other damages to the client's huge facility, the quake caused major ruptured overhead sprinkler pipes that flooded over 10,000 square feet of office cubicle space. Emergency contracts that had been negotiated with key suppliers reduced the estimated repair time of six days down to three. Similar emergency contracts with vendors were in place at The Weather Channel when I worked there several years ago, and this was a company that rightfully prides itself on excellent contingency planning for weather-related disasters.
2. Expect to have key personnel unavailable during a crisis. Three separate disasters recently occurred to clients for whom I worked. These included an earthquake, a train derailment, and a hurricane. In each of these events, key personnel who normally would have enacted disaster recovery plans, were unavailable. They were either at home taking care of family emergencies, or unable to make it in to work due to freeway closures or evacuation orders. Fortunately, in each case there was a backup person who could step in and perform the required tasks. Many disaster recovery plans do not include such contingencies but need to provide them.
3. Provision staff with essentials (flashlights, batteries, phones, water). Three days prior to the expected landfall of Hurricane Charley on Tampa, Florida in August 2004, officials issued an evacuation order for anyone living within a fifty mile radius of the city. The financial services company in California for whom I was working at the time had a major regional business and data center in Tampa. The company had close to a hundred employees in Tampa and directed them to evacuate to safer lodging. The hurricane ended up missing most of Tampa proper, and IT personnel were eager to go back and check on any potential damage to the data center. Upon arriving they found widespread power outages that closed most stores, and a general lack of adequate water supplies. They had not thought to bring simple essential such as flashlights, batteries for cell phones, and water. This became one of the first suggestions for improvement offered during the debriefing sessions.
4. Ensure staff has adequate amounts of petty cash. Many people today rely solely on the convenience of credit, debit or ATM cards for transacting business, payments or purchases. But ATMs sometimes become inoperable due to hardware failures, software viruses, or power outages such as the one described above from Hurricane Charley. This is what happened to the IT staff who came back into Tampa to check on the data center of the financial services company. None of them could purchase essential items for almost two days because all of the ATMs were down and no one had enough cash. Another recommendation that came out of the lessons learned sessions was to ensure support staff had adequate amounts of petty cash to get them through periods of ATM outages.
5. Anticipate last minutes changes in political or environmental factors. Situations associated with a disaster, and the priorities that result from these situation, often change rapidly. Immediately after the California earthquake mentioned previously, the first priority was for the safe evacuation of some 10,000 employees from the huge facility. Many of the employees were stressed from the experience of a major shaker, so tending to their physical and mental wellbeing became the second priority. The third priority was to inspect the physical integrity of the giant structure which involved government building inspectors and corporate facilities managers. Only after addressing these first three priorities was the extent of massive water damage from ruptured sprinkler pipes identified and assessed.

   Another example of a last minute change involved Hurricane Charley. It had been headed directly in the path of Tampa, causing a major evacuation of that city. The financial services company I worked for at the time was headquartered in Southern California and had a large regional center in Tampa. The CIO of the company wanted to send IT specialists his main data center in California to Florida to help the smaller, less experienced staff there prepare the regional data center for evacuation. The CIO was understandably concerned about flying some of his key people into the path of a hurricane, so he decided to have his people land in the safer city of Orlando, some 80 miles northeast of Tampa.

   Just before the hurricane reached Tampa, and just as the team from California was landing in Orlando, the hurricane made a sharp turn to the right and headed toward Orlando. Tampa was spared of any damage from the storm and Orlando now became its target. Fortunately, the storm's energy dissipated before reaching Orlando and thus incurred no serious damage. But the episode shows just how quickly priorities can change.

   These are five important lessons I experienced personally from various disasters. As I mentioned previously, there are other obvious suggestions involving the safety of life, limb and property whenever catastrophic events occur. If I had to sum up the over-riding thought of all these lessons, it would be: plan well ahead and plan ahead well.