IT Management Reference Guide

Most operations managers do a reasonable job at keeping their data centers up and running. Many shops go for years without a experiencing a major outage specifically caused by the physical environment. But the infrequent nature of these types of outages can often lull managers into a false sense of security and lead them to overlook the risks to which they may be exposed. Figure 1 lists the most common of these.

Figure 1 Major Physical Exposures Common to a Data Center

The older the data center, the greater these exposures become. But the relative newness of a data center does not ensure a shop will not have outages. I have had clients who collectively have experienced at least half of these exposures during the past three years. Many of their data centers were less than 10 years old.

Preventative maintenance, testing, inspections, or any combination of these should occur once a year at a minimum. I have worked with some shops that have annual maintenance contracts in place for their physical facilities, including onsite inspections, but choose not to exercise them. Untested safeguards, un-inspected equipment, undocumented procedures and untrained staff are all invitations to disaster that are easily preventable.

A recent personal experience at a financial services client of mine serves to illustrate the importance of regularly scheduled inspections. One of the most critical departments of this firm was housed in a modern office building owned by another company. I was performing business continuity work and was included to be alerted of any unusual occurrences of business disruption. One day around 2:30pm I received a call saying that sprinklers had activated on the first floor of this building over the area where the department in question was located.

I responded to the scene and found the building had been evacuated in an orderly manner per procedure and that the fire department was already onsite. They discovered that a faulty sprinkler head had been accidentally bumped and broken off. What was most surprising to me was that the water that ejected - some 600 gallons worth – was not of the pure, clear variety you might expect. It was filled with black, sooty particles. It turned out that the sprinkler system had not been inspected since its installation some four years prior. During this time it had had accumulated huge amounts of rust, corrosion and other particulate matter. This is what spewed down over a 500 square foot area of open cubicles, much to the surprise of the 18 employees who were sitting under it at the time. Needless to say, regular inspections were immediately intiated.

Tips to Minimize Data Center Exposures

There are a number of simple actions that can be taken to minimize data center exposures (shown in Figure 2). Establishing good relationships with key support departments such as the facilities department and local government inspecting agencies can help keep maintenance and expansion plans on schedule. This can also lead to a greater understanding of what the infrastructure group can do to enable both of these agencies to better serve the IT department.

Figure 2 Tips to Improve Facilities Management

Video cameras have been around for a long time to enhance and streamline physical security, but their condition is occasionally overlooked. Cameras must be checked periodically to make sure that the recording and playback mechanism is in good shape and that the tape is of sufficient quality to ensure reasonably good playback.

Environmental recording device also must be checked periodically. Many of these devices are quite sophisticated; they collect a wealth of data about temperature, humidity, purity of air, hazardous vapors, and other environmental measurements. The data is only as valuable as the effort expended to analyze it for trends, patterns, and relationships. A reasonably thorough analysis should be done on this type of data quarterly.

In my experience, most shops do a good job of periodically testing their backup electrical systems such as UPS, batteries, generators, and power distribution units (PDUs), but not such a good job on fire detection and suppression systems. This is partly due to the huge capital investment companies make into their electrical backup systems—managers want to ensure a good return on such a sizable outlay of cash. Maintenance contracts for these systems frequently include inspection and testing, at least at the outset. However, this is seldom the case with fire detection and suppression systems. Infrastructure personnel need to be proactive in this regard by insisting on regularly scheduled inspection and maintenance of these systems, as well as up-to-date evacuation plans.

One of the simplest actions to take to improve a computer center's physical environment is to remove all tripping hazards. While this sounds simple and straightforward, it is often neglected in favor of equipment moves, hardware upgrades, network expansions, general construction, and—one of the most common of all—temporary cabling that ends up being semi-permanent. This is not only unsightly and inefficient; it can be outright dangerous as physical injuries become a real possibility. Operators and other occupants of the computer center should be trained and authorized to keep the environment efficient, orderly, and safe.

The final tip is to make sure the staff is trained and practiced on earthquake preparedness, particularly in geographic areas most prone to this type of disaster. Common practices such as anchoring equipment, latching cabinets, and properly storing materials should be verified by qualified individuals several times per year.

A Word about Efficiency and Effectiveness

In addition to ensuring a stabile physical environment, the facilities management process owner has another responsibility that is sometimes overlooked. The process owner must ensure efficiencies are designed into the physical layout of the computer facility. A stable and reliable operating environment will result in an effective data center. Well-planned physical layouts will result in an efficient one. Analyzing the physical steps that operators take to load and unload printers, to relocate tapes, to monitor consoles, and to perform other routine physical tasks can result in a well-designed floor plan that minimizes time and motion and maximizes efficiency.

One other point to consider in this regard is the likelihood of expansion. Physical computer centers, not unlike IT itself, are an ever changing entity. Factoring in future expansion due to capacity upgrades, possible mergers, or departmental reorganizations can assist in keeping current floor plans efficient in the future.

References

Schiesser, Rich, IT Systems Management, Prentice Hall, 2002