Are you ready for robots swapping out high-density servers immersed in fluid?
TMGcore believes the market is indeed ready for its OTTO data center platform, which is launching at the SC19 conference in Denver. The two-year old company has been refining its technology in a Dallas-area data center, test-driving its two-phase immersion cooling with bitcoin mining hardware, and developing custom servers, a series of micro-data center enclosures, and a robotic system to replace servers.
“We’ve redefined what a data center looks like,” said John-David Enright, the CEO of TMGcore, which is partnering with Vertiv, Jabil, Corgan, 3M and Dell Technologies to bring its offering to the data center market.
TMGcore says the OTTO platform offers extreme density and efficiency that can radically reduce the space and cost to deploy IT infrastructure.
The OTTO system is perhaps the most ambitious effort yet to create an autonomous data center that can be managed by software and robotics. Enright says it is a response to powerful new hardware that will be difficult to cool with traditional air cooling.
“It became very apparent to us that there’s a big problem,” said Enright. “It’s becoming more expensive and difficult to maintain the amount of compute that companies need to support their business. It’s going to become a game-stopper that will halt innovation and stifle growth.”
Extreme Density, Extremely Small Footprint
The OTTO system brings together multiple forward-leaning technologies to create its platform. These include:
- A micro data center module that can be deployed in three form factors – 60kW, 120kW and 600kW. The OTTO enclosures can be stacked and operate indoors or outdoors, The system can be expanded in increments to manage capital.
- Two-phase immersion cooling, in which servers are immersed in coolant fluid that boils off as the chips generate heat, removing the heat as it changes from liquid to vapor. A closed-loop water system is used to condense the fluid and return it to the tank.
- Custom servers that adapt OEM components in an immersion-ready chassis that can plug into a backplane providing network and power.
- Robotics technology that can “hot swap” servers, removing a failed server from the immersion bath and replacing it with a fresh server.
- A software platform to manage all elements of the data center.
It’s important to note that OTTO is not just a concept. TMGcore has a 120kW micro data center on display on the SC19 expo floor at the Colorado Convention Center in Denver, and says it expects to deliver the first units in the first quarter of 2020.
John-David Enright, the CEO of TMGcore.
TMGcore says OTTO can offer significant improvements in energy efficiency and economics. The company says the 600kW version of OTTO has operated at a PUE (Power Usage Effectiveness, a key efficiency metric) of 1.028 in its Dallas data center, and can be deployed at a cost of $3.2 million a megawatt, about half the current deployment cost of the largest data center REITs.
Those data points are based on racks filled with OTTOblade servers, which pack 6kW of compute power into a single 1U blade. Since the 600kW unit has a footprint of just 160 square feet, TMGcore says OTTO offers the potential to cut operational costs by 80 percent while receiving 10 times more processing power per square foot.
“Today companies are seeking more data processing, higher densities and quicker deployments, yet the available land has become increasing scarce and cost prohibitive,” said Enright. “Users don’t have the space to add the additional data centers to handle the increasing demand.
“We are scaling down the size of the traditional data center without decreasing the processing power,” he said. “We’ve partnered with Corgan to offer clients a solution that can be significantly smaller for the same or greater compute capacity and still have the room within to scale up as the needs increase.”
As TMGcore’s architect, Corgan will work with users to design OTTO installations around a client’s facilities, taking advantage of smaller footprints and space that can’t support traditional air cooling.
Enright says TMGcore is also in discussions with large data center developers about using OTTO as their immersion cooling platform. “We would like to work with companies in the data center industry,” said Enright.
A Look at the OTTO Platform
There’s lots going on in the OTTO platform. Some of its features represent refinements of existing data center technologies. Immersion cooling and modular data centers have both been used widely.
Robots Swapping Servers
The most novel innovation is the use of a robotics system to remove failed servers and replace them with new ones. This ability to swap out servers creates new possibilities in the industry’s long-running effort to create a “lights out” unmanned data center.
Robotic systems have been used for years to manage libraries of storage tapes, with mechanical arms extracting tapes. Google uses robotic systems to destroy old hard drives. Several providers have used robots to automate network connections. But TMGcore is breaking new ground.
“We know that hot-swapping servers has never been done publicly by robots,” said Enright. “We had to develop a pick system between the robot arm and our blade chassis.”
TMGcore worked with Olympus Controls, which creates robotic systems for factory automation, and adapted its technology for physical server management. “I’m a big fan of integrate first, innovate second and invent last,” said Enright. “We did all of that in this process.”
The OTTO robotic arm can latch onto a server, lift it out of the immersion tank, and place it into an enclosure next to the tank that houses backup servers and has open slots where the robotic arm can place the faulty server. The system then grabs one of the backup servers, lifts it into the tank, and pops it into the plug-n-play backplane.
Two-Phase Immersion Cooling
While the robot server butler provides the cool factor, the OTTO system is built atop its ability to cool high-density servers. The initial focus for the TMGcore was ASICs (Application Specific Integrated Circuits) used for bitcoin and other cryptocurrencies.
“We figured that if we can cool that, we can cool anything,” said Enright, who said bitcoin was an ideal test case, but the endgame was always cooling CPU and GPU workloads. “This was all internal. We can make money in Bitcoin mining, and we were able to fund our operations.”
TMGcore’s Everest platform for bitcoin mining got the attention of former Sun CEO Scott McNealy, who became an advisor to the company. “TMGcore is developing one of the most energy-efficient mining rigs, utilizing cutting edge technology and innovative mining boards to dramatically decrease cooling costs and increase token outputs,” said McNealy.
TMGcore uses fluids from 3M and Solvay in its immersion systems, which allows it to match the best fluid to specific workloads.
“There are a number of different applications, and the chemical structures of the fluids are different,” said Enright. “Each have their benefit and place in our world. We designed OTTO to be fluid agnostic. The fluids interact with the compute in a very particular way, so we developed our own blades.”
Immersion-optimized Servers
Server hardware typically has to be adapted for use in liquid immersion. TMGcore decided to design its own line of HydroBlade servers with a chassis optimized for its system, validating “OTTO Ready” designs, boards and components from server vendors.
An OTTOblade server, optimized for immersion. (Photo: TMGcore)
At SC19, TMGcore is showcasing the OTTOblade G1611, a 1U server featuring dual Intel Xeon sockets and 16 NVIDIA V100 GPUs, packing 6 kW into a 1U form factor. Also on display will be an OTTO-ready Dell EMC PowerEdge C4140, running at about 1.8kW of power.
“We’ve seen demand for solutions that can help organizations reduce their data center power consumption,” said Ron Pugh, vice president and general manager, Dell Technologies Embedded & Edge Solutions. “Our team adapted the Dell EMC PowerEdge C4140 for submersion in order for TMGcore to offer a unique solution that diminishes the environmental impact, like lowering cooling costs, without decreasing processing power.”
The OTTO servers will be built and serviced by Jabil, which has a long history in engineering servers for leading equipment vendors.
“When it comes to purchasing data center platforms, time is a huge factor in the decision,” said Ed Bailey, CTO, Jabil Cloud, Compute and Storage division. “Jabil’s scale, manufacturing and integration capabilities combined with TMGCore’s technology will enable TMGcore to accelerate the time-to-deployment which, in turn, accelerates their customers’ time-to-revenue.”
Micro Modular Data Centers
The micro data center enclosures for the OTTO platform will be manufactured by Vertiv, a marquee name in mission-critical power and cooling which has been an active player in modular data centers as part of its edge computing initiative.
An illustration of how an OTTO 60kW edge unit could be deployed in unused space in an office environment. (Image: TMGcore)
“High-density, lower-cost and self-contained data center platforms are vital for our industry’s expansion and are more cost effective and environmentally friendly than traditional data center builds,” said John Schneider, senior vice president of global thermal management at Vertiv. “The Vertiv and TMGcore teams are positioned to provide the new standard for the way data center platforms are designed, built, operated and cooled. The scalability of these new platforms will bring customizable, impactful solutions to companies around the world.”
Each module will require power and network connections, as well as water for the condenser, which uses a closed loop that can either be charged up front or use a water supply.
Inside the module, each immersion tank features a busbar for power, which is supported by 60kW hydro power blade rectifiers. An OTTO tank features 12 1U equipment slots, holding 10 1U servers and two power blades, supporting dual A and B power.
“The requirements are very simple, and it’s smaller and easier than creating an entire air-cooled data center,” said Enright, who said a modular OTTO system can cool 10 megawatts of capacity in about 8,000 square feet, compared to about 90,000 square feet for a wholesale provider.
The Path to Market, and the Road Ahead
TMGcore developed its technology in a data center in Plano, Texas built by Skybox Data Centers. TMGcore ran 5 megawatts of bitcoin mining on its Everest platform in the 105,000 square foot building, which can support up to 35 megawatts of capacity.
In addition to the returns on its bitcoin mining, TMGcore has funded its business with investor support from private family offices, high net worth individuals, and several senior executives from companies in the data center space, Enright said.
“We’ve been well funded and well supported,” he said. “There’s been a lot of interest from the market, especially private equity firms.”
After two years of development and months of preparation for this week’s launch, Enright is excited about the potential of TMGcore’s platform.
“OTTO provides companies with a solution that is scalable, quick to market, secure and extremely energy efficient while remaining cost effective,” he said.
