For many IT companies, the idea of liquid cooling is something akin to throwing an electric toaster in a bathtub. Companies are reluctant to use liquids to cool their servers for fear that it may damage their expensive electronics. Even if they understand that liquid cooling systems are safe to use, they worry about the potential high costs of installing, operating, and maintaining such a system.
But liquid cooling is gaining traction, particularly as hardware for high performance computing (HPC) and artificial intelligence (AI) are producing higher heat densities, often beyond what traditional air cooling is capable of handling.
LiquidCool Solutions is among the vendors that are targeting this segment of the IT market. The company has developed a total immersion cooling system that is safe to use, and highly-effective at removing high heat loads from electronics. Based in Bloomington, Minnesota, LiquidCool has an IP portfolio that includes 30 issued and 17 pending patents for immersive cooling.
“Our technology solves the problems that until now have made total immersion cooling inconvenient and impractical,” says Herb Zien, co-chairman of LiquidCool Solutions, who recently stepped aside as CEO to allow Darwin P. Kauffman, formerly of Seagate, to take over. “We’ve addressed the issues of how to deploy liquid cooling on a large scale, and developed a solution that is safe, systematic, reliable, and cost effective for data center owners.”
Other players targeting the high-density niche with liquid cooling solutions include Green Revolution Cooling, Asetek, CoolIT, Ebullient, Asperitas, Aquila Systems and Advanced Computer Cooling. The companies use a variety of approaches, including cooling fluid immersion, water to the chip and cold plates. Several large installations have also adopted rear-door chilling units from vendors like Motivair and Vertiv to cool high-density cabinets.
How It Works
In LiquidCool’s solution, off-the shelf server components are sealed inside an aluminum leak-proof case. The case is filled with dielectric fluid that is safe to use with electronics, completely submerging the server components inside. The fluid is pumped in from an external coolant distribution unit (CDU), entering the sealed case through dripless quick-disconnect hoses.
The liquid coolant enters at about 115°F, circulates through the server case, and returns to the CDU at about 135°F. The heat removed from the server components can be ejected from the building via a dry cooler, or reclaimed for other use, such as heating the data center itself.
The front view of a rack of servers using the LiquidCool immersion cooling system, with a side view of the sealed cases that house the components and dielectric cooling fluid. (Photo: LiquidCool Solutions)
LiquidCool’s server cases are designed to fit into a standard 42U or 48U rack that has been fitted with a liquid cooling manifold in the back. A 1U Liquid Submerged Server (LSS) carries 1 node. A 42U rack can hold 64 LSS servers, or 8 per shelf, while a 48U rack can hold 72 LSS servers. LiquidCool also offers a 2U Submerged Cloud Server (SCS), which carries 4 nodes. A standard 42U rack holds 21 SCS units or 84 nodes, while a 48U rack holds 24 SCS or 96 nodes.
Advantages of Immersive Liquid Cooling
“Liquid cooling is 1,400 times more effective than traditional air cooling,” says Zien. “Our LSS solution captures 95% of the heat energy from high-density server components. It supports cooling for over 50 kW per rack, or at an individual level, for 500-600 watt servers.”
Both the National Renewable Energy Laboratory (NREL) and Lawrence Berkeley National Laboratory have tested LiquidCool’s immersive cooling system, and found it was more effective than air cooling at removing high-capacity heat loads.
Depending on the data center’s needs, the cost of installing LiquidCool’s LSS solution may be equivalent to or slightly higher than the cost of installing an air-cooled system. But when immersive cooling is used exclusively in a data center, it can completely eliminates the need for air cooling. This provides a huge savings in both capital and operational expense, since data center owners no longer need to buy or operate CRACs, CRAHs, or in-row air cooling units to mechanically cool their server rooms.
“In traditional data centers, fans must constantly blast air to keep servers from overheating,” says Zien. “But if a data center uses LSS and no air cooling, we can reduce their cooling power needs by up to 98 percent, and reduce their total energy use by up to 40 percent. With the elimination of rack and server fans, we provide a 15 percent power savings per rack, and also eliminate the heat and noise pollution caused by those fans.”
By eliminating air cooling, LiquidCool’s solution also reduces the need for “white space” by as much as 60 percent. Data center owners don’t need to have wide aisles between server rows, use hot aisle/cold aisle containment, or use raised floors with plenums to deliver air cooling. This allows them to utilize more floor space to fit more servers into the room.
LiquidCool vs. Other Liquid Cooling Solutions
Liquid cooling has been done before, most commonly by using immersion tanks to hold server components submerged in liquid coolant. However, LiquidCool’s solution offers several differences.
“With immersion tanks, you’re immersing the entire server component in liquid coolant, and hoping it will remove enough heat from the heat-producing chips,” said Zien. “With LiquidCool’s ‘directive flow’ technology, we can direct coolant to the hottest components first – the CPU, memory chips, SSDs, and power supplies – to ensure maximum heat removal.”
This diagram shows how LiquidCool’s design brings cooling fluid to the hottest components first to make the most efficient use of the temperature of the coolant. (Image: LiquidCool Solutions)
Zien also said that maintenance for immersion tanks can be messy, since you have to drain out a large amount of oily fluid to work on the server components inside. “With LiquidCool’s rack-based units, you can ‘hot swap’ an LSS unit in less than two minutes,” said Zien. “Once you have it out of the rack, you can drain the liquid coolant into a special container, replace the server components, refill the unit with the same fluid, and return it to the rack in about 15 minutes.”
“Finally, immersion tanks are usually deployed vertically, like a server cabinet lying on its side, which means they take up a lot more floor space. Our solution works in standard horizontal racks, so you can make better use of floor space, and deploy a higher number of servers in a data center.”
The other type of liquid cooling available is called direct liquid cooling, also known as fluid-to-chip cooling, where the liquid coolant runs through tubes that sit inside rack-based servers, removing heat directly from the chips and other heat-producing components. But even here, Herb Zien believes that LiquidCool’s immersive cooling solution is better.
“With direct liquid cooling, you cool the chips inside the servers, but the servers themselves are also producing heat. So while you use a reduced number of air cooling units, you still need some air cooling to cool the servers. With immersive cooling, you completely eliminate the need for air coolers, so you get a bigger cost and energy savings from not having to buy or operate them.”
Reliability of Liquid Cooling
In October, 2016, after testing LiquidCool’s immersive cooling system at their Energy Systems Integration Facility (ESIF) in Colorado, NREL decided to install the solution at their High-Performance Computing Center.
“The system has been running for over a year, and it’s been very reliable,” says Eric Kozubal, Senior Mechanical Engineer (Mechanical and Thermal Engineering Sciences) at NREL. “The LiquidCool solution effectively removes nearly 100% of the heat from servers that we use for high-performance data analysis, and the immersed servers have worked flawlessly for our research purposes. Likewise, the ancillary cooling systems that remove heat from the servers have operated without failure.”
“When you seal server components in a steel chassis and submerge them in dielectric fluid, you protect them from electrostatic discharge, oxidation, corrosion, air pollution, and other hazards of air cooling,” Zien explains. “Your servers give better performance, last longer, and have fewer equipment failures.”
Zien knows it will take time for companies to embrace immersive liquid cooling. He anticipates that data center owners will install the LSS system as part of their refresh cycle, when they replace outdated servers. Once they see how well LSS technology works, they will gradually migrate to it, until after several years, they will have an entirely liquid cooled data center.
“Liquid cooling is a leap of faith for many companies,” says Zien. “But as they come to understand that it is safe, reliable, and far more effective at cooling their servers, and they realize the energy and cost savings they get, they’ll start to adopt liquid cooling on a widespread basis. I think that will be the nail in the coffin for air cooling.”
Robert S. Lindsay is a Seattle-based writer who writes for and about the data center industry. He can be reached at [email protected].
