• About Us
  • Partnership Opportunities
  • Privacy Policy

Data Center Frontier

Charting the future of data centers and cloud computing.

  • Cloud
    • Hyperscale
  • Colo
    • Site Selection
    • Interconnection
  • Energy
    • Sustainability
  • Cooling
  • Technology
    • Internet of Things
    • AI & Machine Learning
    • Edge Computing
    • Virtual Reality
    • Autonomous Cars
    • 5G Wireless
    • Satellites
  • Design
    • Servers
    • Storage
    • Network
  • Voices
  • Podcast
  • White Papers
  • Resources
    • COVID-19
    • Events
    • Newsletter
    • Companies
    • Data Center 101
  • Jobs
You are here: Home / Cooling / HPC Gets Bigger in Texas: 40,000 Servers, Immersed in Coolant

HPC Gets Bigger in Texas: 40,000 Servers, Immersed in Coolant

By Rich Miller - October 11, 2018 Leave a Comment

HPC Gets Bigger in Texas: 40,000 Servers, Immersed in Coolant

A look inside a DUG immersion-cooled data hall, showing the cooling enclosures filled with dieletric fluid and the piping where the cooling loop enters the enclosure. The Skybox deployment will use more than 700 of these tanks. (Photo: DownUnder GeoSolutions)

LinkedinTwitterFacebookSubscribe
Mail

One of the world’s most powerful computing systems is coming to Houston, featuring more than 40,000 servers immersed in liquid coolant. The massive system from DownUnder GeoSolutions (DUG) is expected to deliver 250 petaflops – more computing power than the world’s top supercomputers.

The DUG system will perform cutting-edge computer modeling for energy companies, bringing new levels of precision to oil and gas exploration. It will be housed in the Skybox Datacenters facility in Houston’s Energy Corridor, where DUG has leased 15 megawatts of capacity. The deal, represented by Bennett Data Center Solutions, is the largest colocation transaction in Houston’s history.

The project will deploy more than 720 enclosures using the DUG Cool liquid cooling system, which fully submerges servers in tanks filled with dielectric fluid, an approach that reduces the huge system’s energy usage by about 45 percent compared to traditional air cooling, the company said.

DUG and Skybox expect the data center to be extraordinarily efficient, operating at a Power Usage Effectiveness (PUE) of 1.05, lower than even the most advanced hyperscale cloud providers.

A ‘Revolutionary’ Data Center

Rob Morris, Managing Partner for Skybox, called the DUG system a “revolutionary” data center. “It’s a very custom build using immersion cooling,” said Morris. “DUG has worked a long time refining this design. It’s the largest immersion project that we’re aware of in the data center space.”

At 250 petaflops, the DUG system would exceed the current specs for the world’s most powerful supercomputer, the Summit system at Oak Ridge National Laboratory in Tennessee. Summit led the June Top500 list with a benchmark of 122 petaflops, and has a projected high end of 200 petaflops.

DUG says the 15-megawatt deployment represents the first phase in a long-term project that could eventually become an exascale system. A petaflop is a quadrillion floating point operations per second, while an exaflop would be one quintillion (1,000,000,000,000,000,000) calculations per second.

Immersion Cooling Goes Hyperscale

The new system illustrates the enormous computing power being brought to bear in high-performance computing (HPC). As systems grow more powerful, the designs to power and cool these systems are evolving, featuring specialized hardware and infrastructure. The most powerful HPC systems have often been housed in government or academic facilities, but the Skybox deal shows that commercial systems are also generating demand for massive systems and custom designs.

The DUG system also showcases the growing adoption of liquid cooling for specialized high-density workloads. The vast majority of data centers continue to cool IT equipment using air, while liquid cooling has been used primarily in HPC, which uses more powerful servers that generate more heat. Liquid cooling can manage higher levels of compute density, and offers potential economic benefits by allowing data centers to operate without computer room air conditioning (CRAC) units, chillers or server fans.

The DUG cooling system fully submerges standard HPC servers into specially-designed tanks filled with a dielectric fluid that is non-toxic, biodegradable, and most importantly, doesn’t conduct electricity.

Free Resource from Data Center Frontier White Paper Library

data center water
Solutions to Data Center Water and Power Availability
The major challenges facing the data center industry today are finite water and power resources. This is especially true for large scale operations and particularly the ongoing trend toward hyperscale facilities. Get the new report that explores a new data center cooling method called the StatePoint Liquid Cooling (SPLC)4 system from Nortek Solutions. This report explores a usage case study in Singapore. 
We always respect your privacy and we never sell or rent our list to third parties. By downloading this White Paper you are agreeing to our terms of service. You can opt out at any time.

Get this PDF emailed to you.

The dielectric fluid is cooled and circulated around the hot server components. Heat exchangers are submerged with the computer equipment, meaning that no dielectric fluid ever leaves the tank. A water loop runs through the rooms and to each heat exchanger.

Servers being immersed in coolant fluid in a custom cooling enclosure invented by DownUnder GeoSolutions for high-performance data-crunching for the energy industry. (Photo: DownUnder GeoSolutions)

Servers being immersed in coolant fluid in a custom cooling enclosure invented by DownUnder GeoSolutions for high-performance data-crunching for the energy industry. (Photo: DownUnder GeoSolutions)

The thermal qualities of the fluid used by DUG allows the use of condensed-water chillers rather than refrigeration, saving 25 to 30 percent of total power usage. Removing all server fans, which aren’t needed in a fluid-immersion system, reduces power consumption by a further 20 percent, DUG says.

Global Search Led to the Energy Capital

Although Houston seems a natural choice for an application for the energy industry, DUG considered a wide range of sites before deciding on Skybox.

“This was an exhaustive world-wide search for a purpose-built, optimized data center facility,” said Dr. Matthew Lamont, co-founder of DownUnder Geosolutions. “Houston was a natural choice given the low cost of power and the fact that Skybox had the available infrastructure ready to go. This facility will allow us to install the fastest supercomputer in the world at this time to meet the ever-increasing demand for energy.”

Lamont said the system is expected to be operational by February 2019.

DownUnder GeoSolutions is based in Perth, Australia and is a leading global geosciences company, using powerful computers to sort through mountains of seismic data to produce images of the earth’s geology.

DownUnder GeoSolutions CTO Phil Schwan (left) and co-founder Dr. Matt Lamont in the company's data center in Perth, Australia. (Photo: DownUnder GeoSolutions)

DownUnder GeoSolutions CTO Phil Schwan (left) and co-founder Dr. Matt Lamont in the company’s data center in Perth, Australia. (Photo: DownUnder GeoSolutions)

This type of computer modeling requires enormous computing power, but can save energy companies millions of dollars. The modeling improves the likelihood that an exploratory well may discover oil or gas reservoirs, averting “dry wells” that are expensive and time-consuming.

“This solution really pinpoints the accuracy of where the oil or gas reservoir is located,” said Bryan Bennett, principal of Bennett Data Center Solutions, which consulted with DUG on the project. “That’s really a game changer. All of it comes down to how fast they can compute.”

DUG operates data centers in its home base of Perth, as well, as well as facilities in London, Houston and Kuala Lumpur, Malaysia. The company was founded in 2003 with a focus on advanced technology – including liquid cooling – that has made it a leading global IT provider for energy firms. The new facility will support DUG McCloud, a new cloud service providing on-demand access to the company’s seismic analysis.

“This facility will allow us to install the fastest supercomputer in the world at this time to meet the ever-increasing demand for energy.”
Dr. Matthew Lamont, co-founder of DownUnder GeoSolutions

The Houston data center industry has plenty of experience providing energy companies with high-density colocation space for seismic modeling. Even by those standards, the DUG project is a milestone.

“This is the largest data center transaction in Houston history, and a significant win for the city as it triumphed over several major global cities during the year-long selection process,” said Bennett. “Skybox Houston was selected after an evaluation of pricing models for data center services, power cost and availability, corporate tax structure and speed to market among other site selection criteria.”

Skybox Campus Focused on Power

Skybox Datacenters is based in Dallas and is a joint venture between the investment firm Rugen Street Capital and real estate developer Bandera Ventures. The company focuses on purpose-built facilities and operates in the wholesale market, handling both speculative and build-to-suit projects.

The Skybox team saw an opportunity in the growth of “Big Data” in the energy industry, with a focus on 3D seismic imaging in high-power-density environments. That’s why the company chose the Houston suburbs for its initial project in 2014, a 20-acre campus in Katy, Texas.

“We are honored DUG chose Skybox for this revolutionary data center,” said Morris. “Our proximity to one of the largest substations in Houston and the ability to ramp up quickly were major factors in the decision. There are so many unique aspects about this project, from the sheer size and ultra-low cost of power to the innovative immersion cooling technology.”

The Skybox Houston One data center, where the company recently began offering colocation services as well as wholesale space. (Image: Skybox)

The Skybox Houston One data center. (Image: Skybox)

Skybox offers a “bring your own power” option through a concept known as ESID (short for electrical service identifier) that allows customers to leverage their own corporate buying power with electric utilities, rather than buying it through the data center operator. Customers that choose the Direct ESID option get their own power meter and easements to control utility access via the equipment yard.

From the inception and design of its Houston project, Skybox was thinking big about power.

“We had invested in a larger duct bank in the north side of the building,” said Morris. “We’ve got a 400 megawatt substation there. It gave us the ability to bring an additional 36 megawatts of power into this building.”

Skybox also has experience with immersion cooling, having just worked with a large deployment for a blockchain customer that leased the first building at the company’s Legacy One data center campus in Plano, Texas.

An Exascale Future?

DUG says its 15 megawatt deployment is the first phase of a longer-term expansion in Houston that could create an exascale system, with the power to process 1,000 petaflops of computing power.

The DUG system arrives amid a push by the U.S. government and HPC community to develop exascale systems at three sites – Oak Ridge National Labs, Lawrence Livermore National Labs in California, and the Argonne National Laboratory in Illinois.

The DUG and Summit systems are different animals, and optimized for different workloads. But a comparison of their specs offers perspective on their scale and computing power.

The Summit system resides in 5,600 square feet of raised floor space in Knoxville, housing 4,608 compute nodes in 256 cabinets, with additional cabinets to support its file system, interconnect switches, and system infrastructure nodes. Each node contains multiple IBM Power 9 processors and NVIDIA GPUs, for a total of about 27,000 GPUs and 9,000 CPU processors, which work together to accelerate processing workloads. The Oak Ridge data hall used by Summit can support 13 megawatts of power, but the system has not yet needed more than 11 MW.

By comparison, the DUG system in Houston will span 10,000 square feet of space, use up to 15 megawatts of power, and feature more than 40,000 servers. The details of the server and processor technology will be detailed in upcoming announcements by DUG, which is expected to discuss the new system at the upcoming Society of Exploration Geophysicists 2018 conference in Anaheim, Calif.

What’s clear is that the exascale movement, and the computing requirements that are driving it, will present new opportunities for the commercial data center industry. As powerful applications like DUG’s seismic tools move to cloud delivery models, they will continue to push the frontier of data center infrastructure.

The project team, from left to right: Gordon Kellerman (Skybox), Bryan Bennett (Bennett DCS), Ben Grigg (DUG), Randy Anderson (DUG), Stuart Midgley (DUG), Mark Lommers (DUG), Emily Smyth (DUG), Rob Morris (Skybox).

LinkedinTwitterFacebookSubscribe
Mail

Tagged With: Exascale, Houston, HPC, Liquid Cooling

Newsletters

Stay informed: Get our weekly updates!

Are you a new reader? Follow Data Center Frontier on Twitter or Facebook.

About Rich Miller

I write about the places where the Internet lives, telling the story of data centers and the people who build them. I founded Data Center Knowledge, the data center industry's leading news site. Now I'm exploring the future of cloud computing at Data Center Frontier.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • Facebook
  • Instagram
  • LinkedIn
  • Pinterest
  • Twitter

Voices of the Industry

Making Your Data Center Water Positive

Making Your Data Center Water Positive Cem Candir, CEO of Chemstar WATER, provides insights on new innovations in data center water treatment and gives details on how some hyperscalers are making their data centers water positive.

DCF Spotlight

The COVID-19 Crisis and the Data Center Industry

The COVID-19 pandemic presents strategic challenges for the data center and cloud computing sectors. Data Center Frontier provides a one-stop resource for the latest news and analysis for decision-makers navigating this complex new landscape.

An aerial view of major facilities in Data Center Alley in Ashburn, Virginia. (Image: Loudoun County)

Northern Virginia Data Center Market: The Focal Point for Cloud Growth

The Northern Virginia data center market is seeing a surge in supply and an even bigger surge in demand. Data Center Frontier explores trends, stats and future expectations for the No. 1 data center market in the country.

See More Spotlight Features

White Papers

power quality management

Understanding the Importance of Power Quality in the Data Center

Clean uninterruptible conditioned power is one of a data center’s most critical elements which can have the most immediate impact if any issues occur. This white paper from Powerside examines the equipment, operational characteristics and conditions that can influence various points in the data center power chain, which ultimately can the impact the power quality and operation of the IT equipment.

Get this PDF emailed to you.

We always respect your privacy and we never sell or rent our list to third parties. By downloading this White Paper you are agreeing to our terms of service. You can opt out at any time.

Newsletters

Get the Latest News from Data Center Frontier

Job Listings

RSS Job Openings | Peter Kazella and Associates, Inc

  • Senior Electrical Engineer - Data Center Design Build - Denver, CO
  • Senior Mechanical Engineer - Data Center Design Build - Denver, CO
  • Electrical Commissioning Engineer - Aurora, IL
  • Electrical Commissioning Engineer - Ashburn, VA
  • Electrical QA / QC Specialist - Data Center - Baskerville, VA

See More Jobs

Data Center 101

Data Center 101: Mastering the Basics of the Data Center Industry

Data Center 101: Mastering the Basics of the Data Center Industry

Data Center Frontier, in partnership with Open Spectrum, brings our readers a series that provides an introductory guidebook to the ins and outs of the data center and colocation industry. Think power systems, cooling, solutions, data center contracts and more. The Data Center 101 Special Report series is directed to those new to the industry, or those of our readers who need to brush up on the basics.

  • Data Center Power
  • Data Center Cooling
  • Strategies for Data Center Location
  • Data Center Pricing Negotiating
  • Cloud Computing

See More Data center 101 Topics

About Us

Charting the future of data centers and cloud computing. We write about what’s next for the Internet, and the innovations that will take us there. We tell the story of the digital economy through the data center facilities that power cloud computing and the people who build them. Read more ...
  • Facebook
  • LinkedIn
  • Pinterest
  • Twitter

About Our Founder

Data Center Frontier is edited by Rich Miller, the data center industry’s most experienced journalist. For more than 20 years, Rich has profiled the key role played by data centers in the Internet revolution. Meet the DCF team.

TOPICS

  • 5G Wireless
  • Cloud
  • Colo
  • Connected Cars
  • Cooling
  • Cornerstone
  • Coronavirus
  • Design
  • Edge Computing
  • Energy
  • Executive Roundtable
  • Featured
  • Finance
  • Hyperscale
  • Interconnection
  • Internet of Things
  • Machine Learning
  • Network
  • Podcast
  • Servers
  • Site Selection
  • Social Business
  • Special Reports
  • Storage
  • Sustainability
  • Videos
  • Virtual Reality
  • Voices of the Industry
  • White Paper

Copyright Data Center Frontier LLC © 2021