The Facility Technologist: Planning For Disasters
By Tom Condon, RPA, FMA
Published in the October 2005 issue of Today’s Facility Manager
We have all witnessed the horrific effects of severe weather and its impact on people and facilities—particularly in the Gulf Coast. There are many lessons to be learned from disasters, and technology is one area where facility managers can implement improvements. Taking action can save facilities and, more importantly, lives.
At the outset, if facility managers have the rare luxury of being involved in choosing the site for a new facility, they need to be aware of the ramifications of prospective locations. Every geographical location has risks associated with it. One obvious factor is weather, which poses varying degrees of risk everywhere. For example, I live in Chicago, where hurricanes have never been a problem. But we do have tornados, which are incredibly devastating, albeit in a relatively small area.
In the Midwest, tornado drills, building safe zones, and evacuation sites are all part of a good facility management disaster preparedness plan. One planning source is the Federal Emergency Management Agency (FEMA) Web site, which contains interactive maps that allow users to zoom in on an area and identify hazards such as floods, earthquakes, hurricanes, tornados, and hailstorms.
Decision makers should consider the weather risks carefully before locating a facility. A number of companies will no doubt be relocating away from the Gulf Coast area wishing they’d been more prudent in their initial choice.
Many regions in the U.S. are prone to earthquakes, and these are not confined to the West Coast. Most people do not know there are also several significant earthquake activity zones in the Midwest. These zones are not as active as those in the west, but when they move, they are tremendously powerful. In 1812, the New Madrid fault in Missouri completely submerged a small town during one of the most powerful earthquakes in American history.
Wherever the facility is situated geographically, the locations of critical assets inside the building are crucial. While critical assets can consist of many things, the most common in a facility include IT systems and data.
The location of a data center inside the facility is critical to its ability to survive a disaster. Many people adopt a strange kind of bunker mentality and assume the data center should be located in the basement. This is one of the worst places for a data center because of the tendency of basements to flood. Likewise, the top floor of a building is also a terrible place for critical assets. They should be at the core of the building—not too high, not too low—and away from exterior walls.
Threats to critical assets do not come from just the outside; facility managers must look at building characteristics as well. Critical IT assets should be kept away from any source of water. Data centers should not be located under main plumbing lines or a bathroom. (I have seen an entire data center taken down by one clogged toilet and a sticky valve.) Facility managers should also check the location of HVAC units, whose condensate drains and cooling water can leak.
Once the data center has been located in a suitable place, facility managers need to address the inherent frailty of most building utilities. Power is the most obvious requirement for a data center, so a backup power source will be needed. A generator is a prime choice, but the center will also need sufficient UPS (uninterruptible power supply) capacity to keep systems going during the few seconds it takes for the generator to start up.
Fuel needs to be available for the backup generator, and it is important to consider how much will be necessary in the event of a disaster. It’s also important to plan where the fuel will come from. (Take a look at the blog of a harrowing account of how one New Orleans data center staff made it through Hurricane Katrina without service interruptions by hauling 55 gallon drums of fuel to its generator.) Consider using a multi-fuel generator that can run on gasoline as well as natural or LP gas, thereby broadening the options.
Cooling requirements are another big Achilles heel for data centers. If the data center is cooled by the same system that cools the rest of the facility, a backup generator will not be able to power this system. Make sure there is separate HVAC for the data center that does not require using the chilled water or cooling loop from the building.
Protecting the actual data of an organization is a crucial aspect of disaster planning. It used to be acceptable to make a few backup tapes and put them in the safe in the data center. That’s great if there is a guarantee the facility will still be there in the morning. But today’s unpredictable world dictates that facility managers think about placing that data at another location.
An organization can work with off-site backup companies to store mission critical data through the Internet using encrypted, secure connections. Typically, these facilities are extremely well equipped to survive any emergency and offer a level of protection that cannot be achieved with traditional backups. One of the key advantages of this approach is that the off-site data storage centers are extremely reliable, most with standards of more than 99.9%. Just as old paper files can be kept at an off-site storage facility, organizations can also store data this way. Facility managers can connect to the storage site through the Internet and send data there overnight.
Off-site storage also offers a cure to one of the most critical and often forgotten backup factors: the data is off-site and will be safe even if the facility is destroyed. Many companies that did not transport their backup tapes off-site learned difficult lessons after 9/11 and other disasters.
Another alternative to traditional, on-site software is to use Application Service Providers (ASPs). These services host the organization’s software, allowing it to be accessed via the Internet. This type of service is a consideration for organizations that cannot afford to build a data center that will survive a disaster.
ASPs take off-site data storage to a new level. These systems not only store the data, but also provide all of the hardware required to run the programs. ASPs provide a server and other hardware so the organization has absolutely no IT systems or data at its site. The organization pays a monthly fee and connects to the system through the Internet.
ASPs can be a great solution for a variety of reasons. There is no large up-front investment to purchase hardware and software; the facility manager doesn’t have to manage those systems; and ASPs have super redundant data centers that are far more reliable than the in-house systems of most companies.
In addition to data related to business systems, such as accounting and payroll, many companies are now creating and backing up digital data about their facilities as well. Digital blueprints, maintenance data, and other facility records are often destroyed in disasters, just when they are needed the most. For example, in a fire or building collapse, emergency responders need to know the layout of the building, its structural components, and the locations of hazardous materials. However, this information is too often destroyed before it can be used to help manage the disaster.
A good example of how this is starting to change is in Chicago, which requires owners of all large buildings to supply this information in digital format to the city Office of Emergency Management and Communications. This information is stored in a secure data center that allows floorplans and other critical information to be shared and analyzed during a disaster.
From choosing a site for the facility to storing critical data required for the operation and recovery of that facility, it is crucial to create systems that will keep physical and informational assets secure. Technology offers a variety of ways to plan ahead of time.
Condon, a Facility Technologist and former facility manager, is a contributing author for BOMI Institute’s revised Technologies in Facility Management textbook. He works for System Development Integration, a Chicago, IL-based firm committed to improving the performance, quality, and reliability of client business through technology.