«White Paper STI-100-015 March 2015 Central Europe, Eastern Europe and APAC HEADQUARTERS - Western Europe, Middle East and Russia Server Technology ...»
Managing variable data center rack densities
White Paper STI-100-015
Central Europe, Eastern Europe and APAC
HEADQUARTERS - Western Europe, Middle East and
Russia Server Technology
NORTH AMERICA Africa
Niederlassung Deutschland Room 2301, 23/F, Future Plaza
Server Technology, Inc. Server Technology
Server Technology LLC 111-113 How Ming Street,
1040 Sandhill Drive Fountain Court 42119 Wuppertal Kwun Tong, Hong Kong Reno, NV 89521 2 Victoria Square Germany Direct line: +852 3916 2048 United States Victoria Street Tel: + 49 202 693917 x 0 Fax Line: +852 3916 2002 1.775.284.2000 Tel St. Albans Fax: + 49 202 693917-10 email@example.com 1.775.284.2065 Fax AL1 3TF firstname.lastname@example.org email@example.com United Kingdom www.servertech.com +44 (0) 1727 884676 Tel www.servertechblog.com +44 (0) 1727 220815 Fax firstname.lastname@example.org
YOUR POWER STRATEGY EXPERTS
Current State of Data Center Density In the first years of the 21st century, the data center was most often a mishmash of old mainframes, networking racks, and various server/storage cabinets with no pre-designed structure. Power density at the rack was not a topic of discussion as most were in the 1-3kW range and the power and cooling infrastructure were overbuilt for maximum uptime (see Figure 1). By 2005, the typical data center manager was exposed to increasing rack power due to new technology, especially blade servers. Yet, the new 5kW racks still proved to be only a minor challenge to the power distribution design with some moving to 3-phase circuits, and a more considerable challenge to the cooling design, bringing about the need to pre-design the data center for hot-aisle / cold-aisle configuration, watch for hot-spots, and provide for backup cooling.
Figure 1: Average power density (in kW) per rack (IDC, 2007 et al.)
YOUR POWER STRATEGY EXPERTSManaging variable data center rack densities Now, a decade later, there remains the discussion of the potential for sudden rapid increases in data center power density. Of course, it is very difficult to increase power density in existing data centers, but the continual call for consolidation is leaving this possibility open for new data center construction. Additionally, the cooling strategies are becoming much more efficient and tuned for minimal needs rather than being significantly overbuilt as in the past.
What Exactly is “High Density”?
The term “High-Density” could refer to any number of things including more equipment in a rack, more racks in a data center, or even more applications on a server. For the purposes of this paper, we will concentrate mostly on the increases in total power utilization per rack and increases in equipment count per rack, though we will also touch on other needs in the data center.
The threshold for high power density set forth by Strategic Directions Group (2014) and endorsed by AFCOM was 8kW per rack, with an extreme density defined at 15kW per rack. Intel (2014) pushed the limits with 43kW per rack extending out to 1100 W/ft2 using custom-built narrow racks and free-air cooling at elevated temperatures. This leads us to the conclusion that the term “high-density” will always be relative, no matter what any single group tries to peg it to.
Emerson Network Power reported in fall 2009 that the average power density of their surveyed respondents was 7.4 kW per rack.
These respondents expected this to grow to 12 kW per rack by 2011 and 16.5 kW per rack by 2019. Strangely, by the fall 2014 survey, the average had dropped to 5.83 kW per rack and projection for two years hence had become 8.9 kW. Indeed, this semi-annual survey of the Data Center Users Group (DCUG) shows that the peak was 7.7 kW in 2012 (see Figure 2). Assuming the variation through the years from 2006 to 2014 was simply due to varied respondents, it seems there has been no increase in rack density on average. So why do these same data centers keep expecting the density to rise? And what might be keeping it stable?
High power density is often linked to high equipment density. That is in general, the more equipment you install in a rack, the more power must be delivered to the rack. We have seen this to be true with the immediate popularity of the Server Technology® HDOT ™ cabinet PDU which has become the industry standard for rack equipment density. It is important to understand that the link between high equipment density and high power density is only valuable in some instances. For example, comparing one rack to another with similar equipment or analyzing one rack evolving over time will demonstrate the link between the two forms of density. On the other hand, any single install could be of very low power equipment which fully loads the RU space in the rack, or a small number of blade chassis drawing much higher power.
Managing variable data center rack densities Figure 2: Average power density (in kW) per rack - Emerson DCUG data from 2006 to 2014 power density will increase from previous builds. We may be coming upon the time where we start seeing the DCUG respondent’s projections come true – higher average power densities may finally be at hand.
Average Power Density vs Peak Power Density It is important to understand and differentiate between the average rack power density and maximum rack power density across a data center floor. It is equally important to understand and differentiate between power density that is averaged over time and power density peak within a period of time. We might call the first topic “spatial power density variation” where the average spatial power density is dependent upon size of the data center and is often tied to infrastructure capacities, and the peak spatial power density is dependent on individual components within the system which is tied to the specific design aspects. For example, a 345 W/ft2 data center (based on white space) might be specified by the fact that 1 MW of power and cooling is available for use over 2900 ft2. Using the AFCOM standard rack area of 25 ft2, 116 racks are deployed at an average power load of about 8.6kW each.
The second topic might be called “temporal power density variation” where the average temporal power density is dependent upon regular application loads, and the peak temporal power density is dependent upon sporadic application loads. For example, any given rack, POD, circuit, or data center will have a peak allowable power load. Within this, there will be an average power load over time at each level based on application. Continuing the example of the 8.6kW racks: if the data center simply provisions 8.6kW for each rack, the total peak load will never exceed the data center allowable, but also any single rack will not be allowed to exceed 8.6kW. This is not optimal as experience shows that the average rack load over time will be much less than the peak. This not only results in the
YOUR POWER STRATEGY EXPERTSManaging variable data center rack densities average load of the data center over time being much less than the allowable, but also the peak load of the data center over time being much less than the allowable for the data center. This is because only a few racks will peak at any given time, not the whole of the data center. The final result of this design is that it is a great over-provision of power.
It would be possible then to say that the average of 8.6kW per rack is useless unless you allow individual racks to exceed the 8.6kW.
Thus overprovisioning of some or all racks allows the overall data center to reach toward the allowable peak. The strategies used for power overprovisioning and for maximizing usage to the limits set by electrical codes require continual monitoring during growth stages and will be further considered later in this paper.
Cost Considerations for High Density Deployments In Schneider Electric’s white paper (2014), Choosing the Optimal Data Center Power Density, the conclusion is made that exceeding about 11kW per rack in design capacity has significantly diminishing returns on reductions in cost with additional design and operational complication outweighing those small cost benefits. This is, of course, a generalization which assumes that space costs are minimum. On the flip side, average rack heights have been increasing and 17kW rack PDUs have become more and more popular.
Presumably, these deployments have been determined to be best for these organizations.
Figure 3: Cost per watt vs power density – Schneider Electric (2014)
The problem with many of these types of analyses is that they leave out the most important factor. That is, they omit the devices such as servers, storage, and network gear that are the power load in the data center. The process of arbitrarily dividing power load between a certain number of racks and then analyzing the cost of infrastructure and cooling does not take into account the IT transactions taking place within the data center. These transactions include the IT refresh cycle in which newer equipment is installed to
YOUR POWER STRATEGY EXPERTSManaging variable data center rack densities provide much better performance but with a nontrivial increase in power consumption. They also include variable application loading over time that may be driven by changes in projects, personnel, or market growth. They even include “green” goals such as driving PUE downward by pushing more IT load. These density-based IT transactions will be revisited later in this paper.
High Power Density vs Other Data Center Business Drivers The discussion of increased power density at the rack does not lie in a vacuum. In fact, it may be a cause or an effect of various other changes or improvements in the data center. Virtualization, consolidation, automation, cloud deployment, modularity, free cooling, big data, hot- or cold-aisle containment and any number of other meaningful concepts interrelate to power density through the topics of efficiency, capacity planning, and uptime.
Efficiency and PUE Efficiency of the power system, including the cooling components has been the hottest topic in the data center for the last several years. Calculation of PUE is being performed by the majority of medium to large scale data centers today. To truly maintain an understanding of the efficiency of the data center, the data center manager needs a continuously operating monitoring system such as Server Technology’s SPM (Sentry Power Manager). Through data trending and reporting, power management at the rack PDU, and rack environmental monitoring, data center personnel can keep a close eye on the relationship between efficiency and power density.
Figure 4 shows that the average PUE in the data center has leveled off over the last few years. Adoption rate of PUE measurement continues to hover around the 70% mark. Increasing power densities may push for increased measurement and reduced PUE.
Figure 4: Average PUE – Uptime Institute Surveys (Symposium 2014)