When we started designing our Prineville, Ore., data center as part of the Open Compute Project, we did so with a "less is more" philosophy. We wanted a highly energy efficient, less costly, simpler and more reliable facility that could serve as a model for other data centers. In the end, we built an innovative data center that has:
How did we do this? When we were brainstorming the design, we asked ourselves what we could do differently from traditional data center designs: What can we remove from the system? Can we raise operating temperatures and have the servers survive? Can we increase server delta T and relative humidity operational ranges to make the system much more robust and efficient? Do we need a centralized UPS, PDUs, or chillers?
Our deliberations resulted in a data center design with four primary innovations that resulted from our eliminating:
The traditional data center loses 21% to 27% of its power due to inefficiencies built into the system. Losses enter the system during every stage of power transformation and conversion in a data center:
By eliminating the centralized UPS and the PDUs, and by designing a 94.5% efficient server power supply, the Prineville data center has a total loss of 7.5% (including the 2% transformation loss). We relocated the UPS so it's closer to the server level and eliminates single points of failure upstream from the Open Compute servers in Prineville. Coupling this with the fact that we no longer need to synchronize between the centralized UPS and the PDUs, availability increases from five 9s to six 9s (99.9999%), based on our assumptions.
The Prineville servers get 45 seconds of backup power at full load from a custom 48VDC UPS, stored in cabinets in the data center aisles. Each unit is 56kW or 75kW, 480/277V, 3-phase input. Each cabinet contains 20 sealed VRLA batteries in five strings. The DC power cables connect directly to the server racks.
The UPS is a standby system, not an online system. During normal operation, there is negligible energy loss. The only energy used during normal operation is to trickle charge the battery.
The cabinet has a battery monitoring system, which measures impedance, voltage and temperature.
More electrical efficiency comes from a custom fabricated reactor power panel that delivers 175kW, 480/277V, 3-phase power to the server cabinet level. The reactor power panel:
Locating the data center in the high desert of central Oregon enabled us to innovate the cooling system design. We didn't build a chiller plant, eliminating associated cooling towers, piping, pumps and controls. The innovative system:
In addition, we have the capability to install indirect evaporative cooling if needed.
These strategies enabled us to build a very efficient, inexpensive and reliable facility.
It's energy efficient in both power consumption and water use. Compare the PUE of 1.07 at full load to that of the typical data center, which has a 1.5 PUE (see page 53). Without chillers and cooling towers, the WUE (water use efficiency) of 0.31 liters/kWH is much lower than the national average of 1.0 l/kWH (this value is based on Green Grid data).
By using simpler construction, we improved reliability. Having fewer components means higher reliability as there are fewer parts that can fail. Such a system is also less expensive to build and operate — our capital expenses were lowered by 45%. We'll have a good sense of how much operating expenses we're saving in time once the system is fully online.
Jay Park is the director of data center design at Facebook.