The deployment of high-density racks of equipment is creating power and cooling challenges for many data centers. The server densification trend is intended to create efficiencies in floor space, cabling, and systems management. However, the growth in power density (Watts per U) with each new server generation is causing data centers to limit rack utilization based on their present cooling capacity. Many data centers are in dire need of new cooling solutions to reap the benefits of server densification.
Though there are several cooling solutions in the industry, new cooling solutions that include highly efficient rack enclosures capable of supporting high power and heat loads are coming on the market. One such solution is a water-chilled, closed-loop cooling system. Experts at the Hewlett-Packard Rack and Power Infrastructure Group writes about the topic in the following article:
Data Centers and Water-Chilled, Closed Loop Cooling Systems
This type of a solution incorporates modular fans and air-to-liquid heat exchangers to remove the high levels of heat generated by advanced server and mass storage systems. A water-chilled, closed-loop cooling system allows a data center to add computing power with minimal impact on the facility’s heat load, thus extending the life of the data center.
This article explains the densification trend that is driving the need for direct cooling at the rack level, and it describes the operation and installation considerations of a water-chilled, closed-loop cooling system.
Data center trends
Originally, data centers were designed to support large, water-cooled mainframes that consumed lots of power and generated intense heat in concentrated areas. As enterprise computers evolved, data center designs changed to support racks of multi-processor servers and storage systems that spread the power and cooling requirements over a larger area. Although this trend allowed data centers to scale easier, it created power distribution, cabling, and system management challenges. The emergence of 1U servers and blade servers allowed organizations to consolidate their data center infrastructures, decrease cable clutter, and streamline server management. However, most data centers are having difficulty adjusting to the effect of high-density racks on power and cooling resources.
A fully loaded 42U rack with dual processor (2P) 1U servers and storage drives requires over 12 kilowatts (kW) of power. A 42U rack with 96 half-height BL p-Class blade servers, including six 1U BL p-Class power enclosures, requires 28 kW of power. As data centers try to accommodate more of these high-density racks, they are moving toward high amperage, three-phase infrastructures. Three-phase power is typically more efficient than single-phase power since it provides more than 150% of maximum available power provided by single-phase power.
The consequence of more power is more heat. Virtually all power consumed by rack-mounted equipment is converted to sensible heat, which increases the temperature of the environment. The sensible heat load is typically expressed in BTU/hr, where 1 W equals 3.413 BTU/hr. Therefore, the heat load of each rack can be calculated as follows:
Heat Load = Power × 3.413 BTU/hr per watt
For example, the heat load for a two-processor 1U server is:
577 W × 3.413 BTU/hr/W =1,969 BTU/hr
This means that the heat load of a fully-loaded 42U rack of servers is 82,710 BTU/hr. In the United States, cooling capacity is often expressed in “tons” of refrigeration, which is derived by dividing the sensible heat load by 12,000 BTU/hr per ton. The cooling capacity needed for a fully-loaded rack of two processor servers is:
82,710 BTU/hr ÷ 12,000 BTU/hr per ton = 6.9 tons
Few existing data centers were designed to provide this amount of cooling capacity for a single rack; and few data centers are capable of distributing adequate airflow directly to rows of such racks.
Many data centers limit power consumption and cooling requirements by limiting rack density (utilization). For example, Figure 1 shows the total power capacity and heat load of a fully-loaded rack of two-processor servers. (Enlarge Figures 1 and 2 for a closer look by double clicking on the image.) The figure also shows the number of servers that can be deployed per rack based on the average rack power density of a particular data center.
The reasonable limit of rack power and cooling capacity for a conventional forced-air (HVAC) cooled data center is 8 kW per rack, or 27,300 BTU/hr per rack. For power densities approaching 15 kW per rack, facility planners can use advanced thermal modeling technologies to help determine the best layout of computing rooms and provisioning of cooling resources. For racks requiring more than 15 kW, the latest cooling techniques use a proven medium—water. Water can remove 3,500 times the amount of heat that an equivalent volume of air can remove.
Water-chilled, closed-loop cooling system
A water-chilled, closed-loop cooling system is designed for data centers that have reached the limit of their cooling capability or that need to reduce the effect of high-density racks on their facility. This type of cooling technology supports fully populated high-density racks while eliminating the need to add more facility air conditioning capacity.
Operation:
The rack enclosure contains three fan modules and three heat exchanger modules that slide into a cabinet mounted on the left side of the rack. Each fan module contains a variable-speed circulation fan, and each heat exchanger (HEX) module contains an air-to-water heat transfer device. Each HEX module discharges cold air to the front of the rack via a side portal. Chilled water for the heat exchangers can be provided by the facility’s chilled water system or by a dedicated chilled water unit.
Airflow distribution:
Most server designs use a front-to-back cooling principle. The water-chilled, closed-loop cooling system evenly distributes cold supply air at the front of the rack of equipment (Figure 2). Each server in receives an adequate supply of air, regardless of its position within the rack or the density of the rack. The servers expel warm exhaust air in the rear of the rack. The fan modules channel re-directs the warm air from the rear of the rack into the heat exchanger modules, where the air is re-cooled and then re-circulated to the front of the rack. Any condensation that forms is collected in each heat exchanger module and is carried through a discharge tube to a condensation tray integrated in the base assembly.
For controlled airflow, the rack enclosure must be closed during normal operation. The enclosure has solid front and rear doors, sidewalls, and top and bottom covers. The front and back doors must be kept closed to ensure that the maximum amount of the cool air is retained within the system. All ra
ck space must be filled by equipment or enclosed by blanking panels so that the cool air is routed exclusively through the equipment and cannot bypass through or around the rack.
Water circulation:
Chilled water for the heat exchanger is regulated by the water group controller, a module that contains a magnetic solenoid valve, check valve, flow meter, and condensate pump. The water group is connected to the facility’s chilled water system (or to a dedicated chiller unit) with flexible 33.8-inch (860-mm) long inlet and outlet hoses. The condensate drain hose, overflow hose, and main inlet and outlet hoses can be routed through the back of the cabinet or downward into a raised tile floor. The inlet and outlet hoses are terminated with quick-connect couplings.
Management module:
For additional control, web-based capabilities to set, monitor, and control temperatures are easily set via an RJ-45 connector in the patch panel. The management module controls the water flow and fan speed to provide the needed cooling capacity and desired server inlet temperature as set by parameters in the web interface. The system maintains the temperature of the server intake air by opening and closing the solenoid-actuated water valve. The valve opens when the sever intake air temperature goes above the set point, and it closes when the air temperature falls below the set point, minus the Hysteresis Value3. The system controls airflow by adjusting the speed of each fan module to maintain the server exhaust temperatures at the appropriate levels. The management module can be configured to send alert traps to network management systems and other SNMP management applications if an alarm condition is detected.
Network monitoring, control, and feedback capabilities are provided through an operator display on the outside of the front door and the RJ-45 network connector in the patch panel. When the management module issues an alarm or warning, the message is shown on the operator display, as well as on an alarms menu and alarm history menu in the web interface.
Cooling capacity:
A water-chilled, closed-loop cooling system rack requires approximately 1½ times the width and 1¼ times the depth of a standard server rack (to allow for the fan and heat exchanger modules and front and rear airflow). However, this type of enclosure has enough cooling capacity to support the heat load of a rack of equipment consuming 30 kW. This heat load is equivalent to that generated by three 10-kW racks, yet the water-chilled, closed-loop cooling system rack occupies 40 percent less floor space than three standard racks. Likewise, the system supports a heat load equivalent to 3.75 8-kW racks (30 kW/8 kW per rack = 3.75 racks) while occupying 65% less floor space and reducing the overall heat load on the facility.
Summary
The design of a water-chilled, closed-loop cooling system can extend the life and capacity of data centers with limited cooling resources. It can integrate with existing and future server cabinets and does not affect how servers are currently deployed, operated, and maintained. The water-chilled, closed-loop cooling system:
• Provides a path for customers to increase power density up to 30 kW per rack
• Supports fully populated high-density racks while reducing the overall heat load on the facility
• Saves valuable floor space and cooling resources that would be required for under-utilized racks
