HPC Gets Bigger in Texas: 40,000 Servers, Immersed in Coolant

One of the world’s most powerful computing systems is coming to Houston, featuring more than 40,000 servers immersed in liquid coolant. The massive system from DownUnder GeoSolutions (DUG) is expected to deliver 250 petaflops – more computing power than the world’s top supercomputers.

The DUG system will perform cutting-edge computer modeling for energy companies, bringing new levels of precision to oil and gas exploration. It will be housed in the Skybox Datacenters facility in Houston’s Energy Corridor, where DUG has leased 15 megawatts of capacity. The deal, represented by Bennett Data Center Solutions, is the largest colocation transaction in Houston’s history.

The project will deploy more than 720 enclosures using the DUG Cool liquid cooling system, which fully submerges servers in tanks filled with dielectric fluid, an approach that reduces the huge system’s energy usage by about 45 percent compared to traditional air cooling, the company said.

DUG and Skybox expect the data center to be extraordinarily efficient, operating at a Power Usage Effectiveness (PUE) of 1.05, lower than even the most advanced hyperscale cloud providers.

A ‘Revolutionary’ Data Center

Rob Morris, Managing Partner for Skybox, called the DUG system a “revolutionary” data center. “It’s a very custom build using immersion cooling,” said Morris. “DUG has worked a long time refining this design. It’s the largest immersion project that we’re aware of in the data center space.”

At 250 petaflops, the DUG system would exceed the current specs for the world’s most powerful supercomputer, the Summit system at Oak Ridge National Laboratory in Tennessee. Summit led the June Top500 list with a benchmark of 122 petaflops, and has a projected high end of 200 petaflops.

DUG says the 15-megawatt deployment represents the first phase in a long-term project that could eventually become an exascale system. A petaflop is a quadrillion floating point operations per second, while an exaflop would be one quintillion (1,000,000,000,000,000,000) calculations per second.

Immersion Cooling Goes Hyperscale

The new system illustrates the enormous computing power being brought to bear in high-performance computing (HPC). As systems grow more powerful, the designs to power and cool these systems are evolving, featuring specialized hardware and infrastructure. The most powerful HPC systems have often been housed in government or academic facilities, but the Skybox deal shows that commercial systems are also generating demand for massive systems and custom designs.

The DUG system also showcases the growing adoption of liquid cooling for specialized high-density workloads. The vast majority of data centers continue to cool IT equipment using air, while liquid cooling has been used primarily in HPC, which uses more powerful servers that generate more heat. Liquid cooling can manage higher levels of compute density, and offers potential economic benefits by allowing data centers to operate without computer room air conditioning (CRAC) units, chillers or server fans.

The DUG cooling system fully submerges standard HPC servers into specially-designed tanks filled with a dielectric fluid that is non-toxic, biodegradable, and most importantly, doesn’t conduct electricity.

The dielectric fluid is cooled and circulated around the hot server components. Heat exchangers are submerged with the computer equipment, meaning that no dielectric fluid ever leaves the tank. A water loop runs through the rooms and to each heat exchanger.

Servers being immersed in coolant fluid in a custom cooling enclosure invented by DownUnder GeoSolutions for high-performance data-crunching for the energy industry. (Photo: DownUnder GeoSolutions)

The thermal qualities of the fluid used by DUG allows the use of condensed-water chillers rather than refrigeration, saving 25 to 30 percent of total power usage. Removing all server fans, which aren’t needed in a fluid-immersion system, reduces power consumption by a further 20 percent, DUG says.

Global Search Led to the Energy Capital

Although Houston seems a natural choice for an application for the energy industry, DUG considered a wide range of sites before deciding on Skybox.

“This was an exhaustive world-wide search for a purpose-built, optimized data center facility,” said Dr. Matthew Lamont, co-founder of DownUnder Geosolutions. “Houston was a natural choice given the low cost of power and the fact that Skybox had the available infrastructure ready to go. This facility will allow us to install the fastest supercomputer in the world at this time to meet the ever-increasing demand for energy.”

Lamont said the system is expected to be operational by February 2019.

DownUnder GeoSolutions is based in Perth, Australia and is a leading global geosciences company, using powerful computers to sort through mountains of seismic data to produce images of the earth’s geology.

DownUnder GeoSolutions CTO Phil Schwan (left) and co-founder Dr. Matt Lamont in the company’s data center in Perth, Australia. (Photo: DownUnder GeoSolutions)

This type of computer modeling requires enormous computing power, but can save energy companies millions of dollars. The modeling improves the likelihood that an exploratory well may discover oil or gas reservoirs, averting “dry wells” that are expensive and time-consuming.

“This solution really pinpoints the accuracy of where the oil or gas reservoir is located,” said Bryan Bennett, principal of Bennett Data Center Solutions, which consulted with DUG on the project. “That’s really a game changer. All of it comes down to how fast they can compute.”

DUG operates data centers in its home base of Perth, as well, as well as facilities in London, Houston and Kuala Lumpur, Malaysia. The company was founded in 2003 with a focus on advanced technology – including liquid cooling – that has made it a leading global IT provider for energy firms. The new facility will support DUG McCloud, a new cloud service providing on-demand access to the company’s seismic analysis.

“This facility will allow us to install the fastest supercomputer in the world at this time to meet the ever-increasing demand for energy.”
Dr. Matthew Lamont, co-founder of DownUnder GeoSolutions

The Houston data center industry has plenty of experience providing energy companies with high-density colocation space for seismic modeling. Even by those standards, the DUG project is a milestone.

“This is the largest data center transaction in Houston history, and a significant win for the city as it triumphed over several major global cities during the year-long selection process,” said Bennett. “Skybox Houston was selected after an evaluation of pricing models for data center services, power cost and availability, corporate tax structure and speed to market among other site selection criteria.”

Skybox Campus Focused on Power

Skybox Datacenters is based in Dallas and is a joint venture between the investment firm Rugen Street Capital and real estate developer Bandera Ventures. The company focuses on purpose-built facilities and operates in the wholesale market, handling both speculative and build-to-suit projects.

The Skybox team saw an opportunity in the growth of “Big Data” in the energy industry, with a focus on 3D seismic imaging in high-power-density environments. That’s why the company chose the Houston suburbs for its initial project in 2014, a 20-acre campus in Katy, Texas.

“We are honored DUG chose Skybox for this revolutionary data center,” said Morris. “Our proximity to one of the largest substations in Houston and the ability to ramp up quickly were major factors in the decision. There are so many unique aspects about this project, from the sheer size and ultra-low cost of power to the innovative immersion cooling technology.”

The Skybox Houston One data center. (Image: Skybox)

Skybox offers a “bring your own power” option through a concept known as ESID (short for electrical service identifier) that allows customers to leverage their own corporate buying power with electric utilities, rather than buying it through the data center operator. Customers that choose the Direct ESID option get their own power meter and easements to control utility access via the equipment yard.

From the inception and design of its Houston project, Skybox was thinking big about power.

“We had invested in a larger duct bank in the north side of the building,” said Morris. “We’ve got a 400 megawatt substation there. It gave us the ability to bring an additional 36 megawatts of power into this building.”

Skybox also has experience with immersion cooling, having just worked with a large deployment for a blockchain customer that leased the first building at the company’s Legacy One data center campus in Plano, Texas.

An Exascale Future?

DUG says its 15 megawatt deployment is the first phase of a longer-term expansion in Houston that could create an exascale system, with the power to process 1,000 petaflops of computing power.

The DUG system arrives amid a push by the U.S. government and HPC community to develop exascale systems at three sites – Oak Ridge National Labs, Lawrence Livermore National Labs in California, and the Argonne National Laboratory in Illinois.

The DUG and Summit systems are different animals, and optimized for different workloads. But a comparison of their specs offers perspective on their scale and computing power.

The Summit system resides in 5,600 square feet of raised floor space in Knoxville, housing 4,608 compute nodes in 256 cabinets, with additional cabinets to support its file system, interconnect switches, and system infrastructure nodes. Each node contains multiple IBM Power 9 processors and NVIDIA GPUs, for a total of about 27,000 GPUs and 9,000 CPU processors, which work together to accelerate processing workloads. The Oak Ridge data hall used by Summit can support 13 megawatts of power, but the system has not yet needed more than 11 MW.

By comparison, the DUG system in Houston will span 10,000 square feet of space, use up to 15 megawatts of power, and feature more than 40,000 servers. The details of the server and processor technology will be detailed in upcoming announcements by DUG, which is expected to discuss the new system at the upcoming Society of Exploration Geophysicists 2018 conference in Anaheim, Calif.

What’s clear is that the exascale movement, and the computing requirements that are driving it, will present new opportunities for the commercial data center industry. As powerful applications like DUG’s seismic tools move to cloud delivery models, they will continue to push the frontier of data center infrastructure.

The project team, from left to right: Gordon Kellerman (Skybox), Bryan Bennett (Bennett DCS), Ben Grigg (DUG), Randy Anderson (DUG), Stuart Midgley (DUG), Mark Lommers (DUG), Emily Smyth (DUG), Rob Morris (Skybox).