By Daniel McGinn
With Super Bowl LI around the corner, maintenance crews and facility management are hard at work preparing Houston’s NRG Stadium for the big game. While there are many crucial areas of preparation for one of America’s largest events, power protection and back-up can fall to the bottom of the list — but it shouldn’t.
Flash back to New Orleans, February 3, 2013. One minute and 38 seconds into the second half of Super Bowl XLVII, half the lights went out inside the Superdome. With 108 million viewers watching from home and another 70,000 in attendance, it may be one of the most infamous power outages in U.S. sports history. Luckily, the backup generators did their job and the game resumed. In the end, it appears an electrical protective relay caused the outage by tripping a circuit breaker.
The best defense against facility outages such as these is having a good offense and being prepared to handle any situation that may arise. There are several steps that can be taken in the short-term and long-term to help ensure uptime and system reliability across facilities and IT.
The Short Game: Tips For Last-minute Prep
While ongoing, preventive maintenance is the best way to ensure the adequacy of backup power systems and other energy reliability factors, there will be circumstances where facility teams only have a short amount of time to get ready for the unexpected. In those cases, it is a good idea to have a last-minute checklist to help prepare for, respond to, mitigate, and recover from an outage. Here are some checklist items to consider:
Check on generators and Uninterruptible Power Supplies (UPSs). While this may be a bit more than a last-minute checklist item, it’ll give you great peace of mind to know that generators will kick in as expected. It’s important to ensure they are adequately fueled and that any critical spare parts are onsite, so repairs can be done as quickly as possible.
Similarly, check on UPS systems and make sure there are no blinking red lights or other tell-tale signs of a failed battery. UPSs and their associated batteries are designed to be durable and dependable; however, maximizing the life of the device requires proper care from the user. Most users are aware that batteries will eventually need replacement, however, many overlook the importance of the battery’s maintenance and don’t realize until it’s too late. One way to avoid last minute surprises is to use remote monitoring services provided by UPS manufacturers that can alert you to existing and pending equipment issues.
If no UPS is installed, seriously consider adding one. A UPS acts as a buffer layer between the grid and the critical load, ensuring that clean, high-quality power is delivered on a consistent basis. They deliver layered system protection from local events and, in the case of utility events like a blackout, provide power during the time it takes systems to properly shut down or generators to come online. It’s the kind of backup insurance you’ll be glad to have when the power and lights go out.
Eliminate communication bottlenecks. In an outage or disaster situation employees and key personnel need access to information such as a list of emergency contacts. The list should include the internal employees and third party vendors who they can contact for various issues during the event. Along the same lines, get in touch with your contacts at designated disaster recovery sites to make sure they have the capacity to handle your IT and equipment loads if need be. During an event such as a Super Bowl, they will likely be experiencing heavy demand, and you don’t want any surprises.
Designate an emergency operations center (EOC). An EOC or “war room” is a site located a reasonable, safe distance from your primary site where key personnel go to focus on the recovery effort. Make it clear to all of these key personnel where the EOC is and when they are expected to leave the primary site and head to the EOC.
Know the effects of water damage to electrical equipment. In the event of flooding or water damage, have an understanding of which equipment must be replaced and that which can be reconditioned. Whether the equipment can be reconditioned, or must be replaced, all services should be performed by qualified personnel who are familiar with the equipment’s operation and construction.
While this checklist is by no means all-encompassing, it’s a good place to start to ensure the basic preparation measures for disaster mitigation are in place.
The Long Game: Take A Pragmatic Approach
The future of electrical system’s health rests in maintenance. A pragmatic approach is best to avoid potential outages while ensuring everything runs smoothly.
One thing for facility management leaders to remember: when determining long-term system reliability, it is important to not confuse the reliability of individual electrical components with the reliability of the electrical system as a whole. Overall system reliability is only as good as its weakest link. Operations personnel can easily be misled if they focus on the reliability statistics of individual components. This false sense of security increases the risk of unplanned downtime.
Develop an electrical emergency action plan. A good starting point would be clarifying “what constitutes an emergency”. From there, identifying and prioritizing critical functions is essential to efficiently and safely restoring power. Emergency service contacts, critical equipment pricing, lead times, and a deployment strategy should also be addressed. The electrical emergency action plan should be incorporated into the organization’s workplace safety policy and routinely communicated to employees.
Look for the optimal blend of maintenance strategies. The key will be to create a mix of tactics that will meet reliability and availability requirements at the lowest cost.
A preventive or “scheduled” maintenance approach is characterized by routinely performed maintenance (regardless of equipment condition). In some cases, maintenance may be unnecessary but is nevertheless performed on a regular time schedule to avoid issues in the future. While more expensive at the outset, preventive maintenance provides long-term ROI by helping personnel avoid the often much costlier damage that can result from one-off reactive maintenance strategies that introduce risk.
Reliability-centered maintenance (RCM), which takes a systems view as opposed to an equipment or a component view, is another approach that prioritizes the maintenance expense on critical vs. noncritical functions and integrates preventive maintenance, predictive testing, and inspection and run-to-fail maintenance strategies to meet business objectives. An ongoing process, RCM gathers performance data to improve equipment design and enables management to make more informed future maintenance decisions.
Maintenance strategies can be aided by implementing technology like building management systems (BMS) that help to monitor building performance by collecting and analyzing data from electrical, HVAC, security, and IT systems to improve reliability and efficiency. Additionally, digital remote monitoring services allow for real-time monitoring of the facility through smart alarming, remote troubleshooting, and visibility into critical IT equipment to reduce downtime and improve efficiency and scale.
Modernize aging electrical gear. As in the case of Super Bowl XLVII in New Orleans, aging electrical equipment can hamper continuity of services and as a result, power reliability. While switchgear may be within its intended “useful life”, it may not be robust enough to support a facility’s evolving technology requirements. Current technology circuit breakers feature enhancements that were not available 15-20 years ago. The good news is that in most cases, there’s no need to rip out all the equipment and start over. A retrofit solution can provide a quality and quick turnaround fix.
Effective planning and having the right technology in place will help to reduce the chances of an outage happening and will expedite recovery so facilities can get back in the game.
McGinn is the director of secure power systems at Schneider Electric. In this role, he is responsible for business development strategies and execution in the area of UPS and related technologies for the industry and infrastructure spaces. McGinn has over 20 years of experience in process automation, energy management, and manufacturing IT systems with extensive experience in the application of UPS technology. He earned a B.S. in electrical engineering from the University of Michigan at Ann Arbor.
