IT Management Reference Guide

During most of 2004 I worked as a business continuity consultant for a major financial services company headquartered in southern California with regional centers nation-wide. I was a member of a small team functioning as the company's business continuity department. One of our first recommendations was to solicit bids for a disaster recovery services contract. Our business justification clearly showed the need for a remote recovery site for both technical and business restorations in the event of a disaster. Executives promptly authorized us to proceed with the process.

We compiled a list of prioritized requirements and incorporated them into a comprehensive and formal request for proposal (RFP). Other necessary departments such as legal and purchasing helped to finalize the document and in early April the company's purchasing agent sent the RFP out to three prospective bidders: IBM, HP and Sungard. The prospective bidders had three weeks to send back their proposals, and with days to spare all three submitted competitive responses. We evaluated the proposals thoroughly, visited their recovery sites and talked with several of their customers. For a variety of reasons including cost, support, references, and capability of facilities, we awarded the contract to Sungard on June 1st.

Our team was eager to test and demonstrate Sungard's capabilities. During July and early August of that year I participated on a cross-functional team planning a simulated disaster recovery exercise. Our plan involved two aspects. One was to restore critical business functions at Sungard's business recovery center in Cypress, California which was some 30 miles from the company's headquarters. The other was to recover technical operations at Sungard's technical recovery center in Scottsdale, Arizona. All planning was proceeding smoothly. On August 12th, with just a week to go before the exercise, our planning efforts took a radical change.

Switching from Simulated Disaster to a Real Disaster

Tropical storm Charley in the Atlantic Ocean had grown into a full blown category 2 hurricane, and it was headed directly for Tampa, FL. My client has a large regional center in Tampa close to the gulf. On Thursday afternoon, August 12th, residents of the entire Tampa region were ordered to evacuate. For the next 72 hours our team suspended all planning for the simulated exercise, and went into full emergency mode to cope with a real-life disaster. We pulled together all necessary personnel, took preventative actions in Tampa to facilitate a recovery, and evaluated our options of re-locating the Tampa office functions to either New York or Chicago.

Even though New York was planning for the Republican convention, and hotel rooms and security were possible concerns, we quickly determined that New York would be the better of the two choices to facilitate a quick and successful recovery. IT, Facilities, Telecom, Administration, Security, Business Continuity and the Business Units all participated, locally and remotely, in the planning and decision-making process.

On Friday morning we met again to report status on various recovery plans that by now involved teams of personnel in southern California, in Tampa, and in New York. Managers also authorized flying in special support personnel to Orlando because they considered Tampa too dangerous a place to land at that time. After all of the technical teams gave their updates, the recovery team leader inquired if there were any other concerns to discuss. The business unit representative raised her hand and quietly asked what soon became a bombshell of a question, "What about the loan documents?"

All 13 of us in the meeting turned to her as the meeting leader responded, "What do you mean, the loan documents?" The business unit rep went on to explain how there were tens of thousands of loan documents in cardboard boxes on the first floor of the building. These documents were of loans that were in process, and by policy would be scanned in as electronic documents only after the loans had fully funded. Because the building was within a quarter mile of the gulf coast and forecasters predicted ten foot storm surges, the team's emphasis quickly switched from the recovery of very high tech systems to the safeguarding of very low tech, but critical, hardcopy records. For the next three hours staffers moved hundreds of cardboard boxes to higher levels and safer locations.

Mother Nature Takes a Turn for the Better

By Friday afternoon we had all of our plans in place and were about to activate them. Over 2 million people had evacuated the Tampa area and the hurricane had grown to a category 4 with winds up to 145 mph. Then, rather suddenly, hurricane Charley turned sharply to the right and made landfall just north of Fort Myers, about 100 miles south of Tampa. Within hours we were able to confirm that my client's Tampa office was relatively unaffected. All of this occurred on Friday the 13th.

We still had follow-up work to do the next day. Teams went in to assess the facility, restore power, and make other arrangements to ensure full operation by Monday morning. Aside from a few minor local phone company problems, there were no significant issues to report. The entire facility was back to full operation by Sunday morning. As we resumed our planning for the simulation exercise that would occur this week, it was hard to ignore the irony of what we had just experienced. We had talked about making our simulation as realistic as possible to be able to assess our emergency preparedness. Formulating contingency plans for a real-life category 4 hurricane directly in your path is about as realistic as one can get. Much was learned from the experience. And fortunately for all involved, the quick and thorough contingency plans never had to be activated.

Lessons Learned

The recovery team learned several important lessons from this preparation. One was the importance of having emergency supplies on hand such as flashlights, batteries and water. The second was the need to have petty cash available. Lengthy, widespread power outages knocked ATMs out of service for up to ten hours in some areas. Most persons relied solely on ATMs or credit cards for purchases and during power outages cash was the only way to buy needed items.

The most important lesson was not to overlook the importance of critical documents. After days of planning, evaluating and addressing every possible technical contingency, the recovery learned that vital hardcopy records turned out to be one of the most crucial of elements. For all of our emphasis on advanced technology, there is still a large portion of corporations that rely on non-technical hardcopy materials and these need to be included in any recovery plan. This was the most significant lesson of all.