Thermal strategies – planning and designing for the next generation of AI data centers

AI workloads are pushing the boundaries of infrastructure and reshaping thermal design requirements. As traditional assumptions are being tested and new cooling challenges emerging, Mihir Nandkeolyar – Director of Technology Strategy, Data Center Thermal Management at Johnson Controls – sat down to discuss the current climate and how thermal strategies have changed in the last few years.

During the Data Centre Magazine webinar “Designing for Density: Thermal Strategies for AI Data Centers,” Nandkeolyar took a deep dive into data center design. In this article, we share some of the highlights related to infrastructure design and the key aspects that organizations should consider when building the next generation of AI Factories.

1. Framing the cooling conversation as simply air vs liquid cooling often misses the bigger picture

The exponential growth of AI and subsequent increase in rack densities has fundamentally changed thermal management and the way that data centers are cooled. Traditional air-cooled data centers were designed around certain assumptions that the industry has been pushed to rethink. Once rack densities climb past 20-30kW, air alone struggles to keep component temperatures within safe operating ranges.

Today, AI workloads drive rack densities to between 50-150kW – and often far beyond that. For this reason, liquid cooling or direct-to-chip liquid cooling have increased in prominence. However, as Nandkeolyar suggests, framing the conversation as simply “liquid vs air cooling” often misses the wider context.

“It’s important to provide the right solution to the right challenge. GPUs today put out so much heat in a very small die – around 50mm by 50mm – and that drives the need to use a different type of cool plate that takes liquid as medium. [However] there are other parts of the data center that haven’t changed all that much – some of the standard networking racks, power supplies and other loads can still be cooled through air.

So, it’s not just ‘my whole data center is liquid or my whole data center is air’ – it’s a hybrid approach. It’s understanding that ratio [between air and liquid cooling] and considering if that ratio is going to be the same as you change the type of power and the type of chips that you use. Do you have the flexibility to float between that air cooling capacity and liquid cooling capacity and the infrastructure to support it?”

2. Resource analysis – a vital first step when scaling AI infrastructure

Throughout the conversation, Nandkeolyar spoke of the importance of resource analysis and having a deep understanding of what’s available to you as you scale upwards. When you are going from a 20MW or 30MW to a 200MW or even 1GW data center, the difference in scale is often vastly underappreciated. The amount of land, power, labor and other resources that are needed scales in a way that can be hard to fathom.

“On paper, adding a zero to the number of megawatts you need seems simple. But getting ten times the amount of power equipment, ten times the amount of cooling equipment and running pipe that moves ten times the amount of thermal mass really starts changing [the scale of things you need to consider].”

When asked what is the first practical step that he’d take if he was an operator who was planning for the next phase of AI infrastructure demand, Nandkeolyar reiterated the importance of resource analysis and undertaking a thorough assessment of existing infrastructure.

“I think it’s really understanding the blueprint that you want your build to be based on and understanding the resources that you have available to you. As you start to study that, you’ll learn that certain things [are less feasible].

For example, a lot of folks will think ‘Oh, I can tap into stream water or another natural body of water for heat rejection instead of pulling from city water.’ As they dive down that resource rabbit hole, they’ll realize that the amount of river water is at such a high magnitude that it really may not be an option. So, really dive deep into all the different resources that you assume you have if you are scaling from a 10-20MW to a GW facility.

Those are the first practical steps – What do you have to work with? How much land do you have to work with? How much power, space, water do you have to work with? Or, ideally, if you can go waterless. Then you can move on to thinking about your priorities and goals.”

3. The key role of collaboration in data center infrastructure design

When building data center facilities, organizations need to consider the rack densities and technologies of today – but also what’s coming down the line. This requires more than forethought. It requires creativity in design and the flexibility to accommodate for the next industry standard. This includes new cooling technologies or leaving enough space for additional coolant flows, new infrastructure, equipment and new ways of extracting power.

Five years ago, liquid cooling wasn’t as commonplace as it is today, and many legacy data centers didn’t incorporate it into their designs. This wasn’t poor planning. It was simply not knowing what the data center of tomorrow would look like. So, when building the data center of the future, organizations need to maintain a certain level of open-mindedness for new innovations and new possibilities.

For Nandkeolyar, collaboration plays a vital role when planning for future innovation.

“It’s less about creativity and more about collaborating with a lot of folks such as chip makers and those who design the coolant racks, the coolant systems that distribute to the racks, the power systems, the busways and the CDUs. It’s important to ask:

• What is possible?
• What are thermal limits?
• What are the power limits?
• If I were to use X amount of power in this row instead of Y, how much space do I need to leave so that I’m not physically limited by my building when I need to upgrade equipment?

And, outside of this, we find that the best way to share information is to have common blueprints that everybody can work from. Everyone is going to change it a little bit. Starting with good, consistent blueprints allows organizations to scale with speed and gain reassurance in knowing that the blueprint provides a generally good way of doing things.”

4. Planning for warmer water temperatures and resource efficiency

Another highlight was an important but often overlooked dimension of AI cooling strategy – the temperature of supply water. During the conversation, Nandkeolyar was asked a pertinent question, “What infrastructure decisions made today will have the biggest long-term impact as AI densities continue to grow?” His thoughts immediately turned to chilled water temperatures.

Nandkeolyar proposed that, from a cooling standpoint, “planning for warmer and warmer water is the number ticket to increased efficiency.” He went on to talk about how the industry often ruminates over how various thermal technologies can inch away at 1% or 2% efficiency gains, but nothing is more efficient than reaching a point where you can switch off the equipment. And being able to switch off the equipment happens more frequently, for more hours, when you can raise the temperature of the chilled water outlet.

Many liquid cooling systems can operate effectively with supply water temperatures of 35 to 45 degrees Celsius – far warmer than the chilled water temperatures used in traditional data center cooling (typically 7 to 12 degrees Celsius.)

“Raising the chilled water temperature means that in places like Europe, where 80% of the hours are below 25 degrees Celsius, you may not need to run chillers [during those lower temperature hours]. You need the chillers for those peak hot days, but raising the supply water temperature means that you get more low-load or low-compressor hours.

Today, you may not have GPUs that work at those temperatures. But planning for that in the system and buying equipment that can support those higher temperatures is probably the best thing you can do in the long-term to support higher densities.”

Designing for density: thermal strategies for AI data centers

The thermal demands of AI workloads represent a genuine inflection point for data center operations. Facility operators who act decisively by assessing their infrastructure, selecting appropriate cooling technologies and building collaborative relationships will be best positioned to support the AI workloads of today and adapt to what’s to come.

You can watch the full webinar here.