Immersion Cooling At Scale: BitFury Pushes Density to 250kW Per Rack
Immersion cooling is having a bit of a breakout moment. Bitcoin specialist BitFury will use liquid cooling technology from 3M to build a 40 megawatt facility that will feature power densities of 250kW per enclosure.
The new facility will be the largest implementation to date of immersion cooling, which houses servers in enclosures filled with fluid, and pushes the boundaries of rack densities.
The new BitFury facility will be fully optimized for immersion cooling, eliminating much of the infrastructure seen in traditional data centers. The extreme densities and efficiency are critical to success in cryptocurrency mining, where profitability has been squeezed by competition and price declines.
Seeking to find an advantage through data center design, BitFury acquired Allied Control, which specializes in two-phase immersion cooling (2PIC) using Novec coolant fluid from 3M.
“With 250 KW per rack and not having reached the limits yet, we now have the ability to grow quickly and spread the cost of the cooling system over many hardware generations,” said Kar-Wing Lau, vice president of operations at Allied Control.
Here’s how the system works: As the chips generate heat, the Novec boils off, removing the heat as it changes from liquid to vapor.
The system is extremely efficient, with a reported Power Usage Effectiveness (PUE) of 1.02. Those extreme PUEs are possible because the facility doesn’t require raised floors or room-level air cooling.
The Novec liquid inside a BitFury cooling enclosure actively boils as it changes phase, removing heat from bitcoin mining hardware. (Image: 3M)
These techniques and specs may seem alien to most data centers, where workloads average about 5kW a rack in enterprise facilities and 8kw to 10kW per rack in hyperscale server farms.
But immersion may find a broader application in high performance computing (HPC) as the U.S. government seeks to create an exascale supercomputing system. Last fall at the SC14 conference, a panel of HPC experts outlined their expectation for a rapid expansion for liquid cooling that may extend beyond its traditional niches. At Data Center Frontier we’re tracking this transition, which reaches a new level of scale with the BitFury announcement.
Rethinking Data Center Cooling
Keeping servers and storage equipment cool is a key function of any data center. Most use cool air in a “hot aisle/cold aisle” with water chilling loops confined to the air handlers and CRACs (computer room air conditioners) housed around the perimeters of the data hall.
The alternative is to bring liquids into the server chassis to cool chips and components. This can be done through enclosed systems featuring pipes and plates, or by immersing servers in fluids. Some vendors integrate water cooling into the rear-door of a rack or cabinet. Immersion takes a different approach, sinking the equipment in liquid to cool the components.
Immersion cooling has been a small but growing niche within liquid cooling, as vendors like Green Revolution and Allied Control built business in the HPC sector, often in academic computing centers or in seismic modeling for the the energy industry. In addition to supporting extreme power density, immersion cooling has the potential to slash the cost of data center infrastructure, allowing users to operate servers without a raised floor, computer room air conditioning (CRAC) units or chillers.
Bitcoin has proven fertile territory for immersion cooling, reflecting a sharper focus on data center efficiency by industrial mining operations, whose profit margins have been squeezed by a plunge in the price of bitcoin. After soaring as high as $1,100 in late 2013, the value of a bitcoin has settled at lower levels, currently trading at about $295. This forced many players in the bitcoin cloud mining space to shut down or halt payouts to customers.
Previous generation BitFury data centers assembled walls of ASIC hardware and used vinyl curtains to segregate hot and cold aisles. (Photo: BitFury)
The survivors have been Chinese mining pools and bitcoin hardware specialists like BitFury, which creates specialized semiconductors known as Application Specific Integrated Circuits (ASICs). Last year BitFury raised $20 million in venture funding to roll out a global data center network, including facilities in Finland, Iceland and the Republic of Georgia.
The new facility in Georgia marks a major change for BitFury, which in the past has used the low-budget “hashing center” designs common to the bitcoin sector, featuring high-density hardware and low-reliability power infrastructure, often housed in former warehouses.
Bringing Scale to Immersion Cooling
The new site shifts the data center design paradigm to Allied Control’s immersion technology. Plans call for a first phase of 40 megawatts of capacity, or about 160 immersion enclosures housing 250kW of IT load.
“The PUE we’ve achieved leads us to the next level of data center cooling for high performance computing,” said Valery Vavilov, BitFury CEO. “We’ve taken many steps to optimize our infrastructure for securing the blockchain and blockchain transaction processing that have led us to many different technological innovations. These experiences and technologies could be useful for next generation data center constructions of high performance computing and supercomputing.”
In addition to supporting extreme power density, immersion cooling has the potential to slash the cost of data center infrastructure, allowing users to operate servers without a raised floor, computer room air conditioning (CRAC) units or chillers. It also allows ASICs to operate without fans, which are typically among the largest components of a bitcoin mining rig.
Immersion cooling tanks also support rapid hardware refresh cycles, which is a particular focus for BitFury, which has frequently updated its ASIC design to keep pace with the arms race in bitcoin hardware.
Over the past five years, 3M’s Novec has been used in server cooling in a series of technology demonstrations, starting with Data Center World in 2012 and then a number of increasingly ambitious deployments. Novec is ideal for immersion cooling because it has a low boiling point (61°C/142°F) and changes states from liquid to vapor as it is heated by server components. The Novec vapor rises, condenses on a water-cooled condenser coil and falls back into the tank without the use of pumps. Here’s a diagram illustrating how the system works:
