This article was adapted from a recent edition of The Data Center Frontier Show podcast. Visit our podcast page to listen to the full episode.
The idea of using robots to automate data center operations has been the object of years science experiments and proof of concepts. In November, a data center startup called TMGcore demonstrated a robotic system that can replace servers housed within an immersion cooling tank.
This ability to swap out servers creates new possibilities in the industry’s long-running effort to create a “lights out” unmanned data center. Which raises a question: Are we on the verge of a new frontier in data center automation?
Today we’ll look at the history of robotics in the data center, look at some implementations, and discuss how they may help shape the data centers of the future.
First, why robots? There are three important reason for the fascination with the use of robots in the data center industry:
- The first is the data center industry’s long tradition of continuous improvement in automation and efficiency, which has intensified as teams of system admins and SREs have had to manage a larger and larger volume of servers.
- Secondly automation has also been advanced as a way to improve uptime by detecting potential problems and addressing them before they manifest and failures and uptime
- The third reason is that robotics has been discussed as a potential tool for addressing the looming workforce shortage in the data center. Some folks believe we are likely to see shortages of skilled data center staff as experienced employees reach retirement age.
It’s hard to talk about AI and robots without noting the public anxiety about the technology becoming self-aware, or ushering in our New Robotic Overlords. It’s not an accident that Americans wonder if artificial intelligence (AI) will turn the tables on its creators and subjugate humanity. It’s a reflex drilled home by decades of blockbuster Hollywood movies like the “Terminator” series or “The Matrix” series or “Alien” or “Westworld” or “Avengers: Age of Ultron” or “Tron” or “Blade Runner” or “War Games” or “I Robot” (Will Smith version) or … well, you get the picture.
Robotic technology for the automotive industry, showcased in a display at Narita Airport in Tokyo. (Photo: Rich Miller)
So it’s important to note that when we talk about robots in the data center, we mean industrial robots, that bear little resemblance to the humanoid Terminators, and are closer to the large mechanical arms you see on factory floors and assembly lines.
Folks have been talking about an automated “lights out” data center for a long time. In 2006 HP announced plans to move to a “Lights Out” model. Five years later, AOL said it had created a small, completely automated, unstaffed data center.
More recently, EdgeConneX has built a series of regional data centers that can operate unmanned, using a remote monitoring, sensors and an advanced “edge operating system” to track operations and deploy “warm hands” staff only when needed.
In 2013, I was talking with Bill Kleyman, who has written for both Data Center Frontier and Data Center Knowledge, which I was editing at that time. Bill loves all kinds of new technology, and was fascinated with the potential for robots to operate in data center environments. So he did a series of stories about the rapidly advancing capabilities of industrial robots created by companies like DevLinks or Japan’s Fanuc. This was one of the first meaningful discussion of how factory robots might be adapted for data center use, including equipment replacement, and the major obstacles that present challenges. So what does the road to data center robots look like?
The History of Robots in the Data Center
Robotics has been used for many years in cold storage operations for older data. My first encounter with a robot in a data center came in 2015. I was touring the massive Facebook data center campus in North Carolina, and got an early look at a new storage system for rarely accessed data, known as cold storage. I walked into a large, mostly dark data hall top a row of racks housing the new cold storage system, which used high capacity Blu-Ray disks to store old photos and status updates. They were housed in large rack-based storage units that hold thousands of Blu-Ray disks. Each rack included a robot retrieval system, which was housed in the bottom of the rack. When data was requested, the robotic arm springs into action, travels along tracks on either side of the rack, fetches a Blu-Ray disk, pulls data off the disc and writes it to a live server.
Even before the Facebook Blu-Ray system, tape archives seen at Google and high-performance computing data centers use robotic arms to locate and retrieve backup storage tapes.
In 2013, IBM got attention when it adapted some models of iRobot Roomba vacuum to monitor temperatures in its data centers.
In 2013, IBM got attention when it adapted some models of iRobot Roomba vacuum to monitor temperatures in its data centers. IBM built a Roomba that moseys around a data center with sensors and a webcam attached, measuring temperature and humidity, and creating maps of their distribution. IBM uses the resulting maps to see where hot spots are developing. They can also scan RFID tags and manage inventory. At the time, IBM was supposedly using these in 9 data centers, but I haven’t seen any subsequent accounts of this being deployed at scale, or that this was anything more than science experiment.
In 2018, Google’s Joe Kava disclosed that the company was using industrial robots to smash used hard drives that were being decommissioned. When a disk can’t be verified as being fully wiped, they shred the hard drives
One of the most interesting applications of robotics has been in interconnection. In Frankfurt the interconnection specialist DE-CIX created a robotic system which it named Patchy McPatchbot, which has automated the provisioning and upgrading of networks. Patchy is a mounted on an Optical Distribution Frame instead of a standard rack, and its robotic arm can plug and unplug connectors, as well as manage cables. Here’s a look at Patchy McPatchbot in action.
There’s also a company called Wave2Wave, which has a robotic optical fiber switch, which also uses robotic arms to manage cross connections. The Rome switch is 19 inches wide and can fit into a standard rack. Wave2Wave has worked with NTT on automating fiber deployments in Japan. The robot knows the position of ports within the chassis, and can manage cables and slack with precision.
So how do we make the transition to robot server management? Interestingly, the front lines of this effort are in the field of immersion cooling, where the servers are dunked into a tank of liquid coolant. I had never thought that much about how immersion could lead to the creation of unmanned data centers powered by AI and robots.
But then I spoke with Scott Noteboom, who many readers may know from his work at Yahoo, where he designed an extremely efficient Compute Coop – a data center that borrowed design concepts from the thermal management of a chicken coop. Scott has been a thought leader in applying AI to managing data center infrastructure, and just joined immersion specialist Submer as its CTO. He has a vision for a future filled with autonomous data centers, which could look very different from the data halls you see today.
Noteboom believes robots and software can create a data center environment completely optimized for machines, with no humans involved. He helped develop AI agents that can monitor the sound of generators, listening for any anomalies that require attention. He points to the highly automated operations of warehouses as an example of what data centers might be able to accomplish. Like warehouses, data centers feature an organized structure, a lot of automation, and a floor plan built to be as efficient as possible.
Now that robots can smash hard drives, and swap out storage media and network connections, the next frontier is servers. We’ll talk about his new frontier after a short break.
What Does the Future Hold for Robots in the Data Center?
Shortly after I spoke to Noteboom, I had a briefing with TMGcore, which has turned that vision into a working automation platform it called OTTO, which is demonstrated on the EXPO floor at this week’s SC19 conference in Denver. John-David Enright, the CEO of TMGcore, shared the details of the company’s technology, which uses two-phase immersion cooling, in which servers are immersed in coolant fluid that boils off as the chips generate heat, removing the heat as it changes from liquid to vapor. Enright and his team designed micro modular data centers to house the immersion tanks, and wanted to create a robotics system to swap the servers.
So TMGcore worked with Olympus Controls, which creates robotic systems for factory automation, and adapted its technology for physical server management. The robotic arm was customized so it can latch onto a server, lift it out of the immersion tank, and place it into an enclosure next to the tank that houses backup servers and has open slots where the robotic arm can place the faulty server. The system then grabs one of the backup servers, lifts it into the tank, and pops it into the plug-n-play backplane that provides the power and fiber connections.
TMGcore has just demonstrated its platform for the first time, so it remains to be seen how the market will embrace the platform. But it poses a question: What is the future of robotics in the data center?
First, it’s important to be clear that there are always going to be humans involved in data centers. Most data centers, anyway. But in the near future, we will begin to see a disconnect between the enormous volume of servers and storage that must be managed, and the volume of skilled workers available to manage them. Robotics will be part of the toolkit that data center industry must use as it seeks to accelerate the automation of cloud
In the unmanned data centers operated by EdgeConneX the automation is driven by sophisticated software, management tools and sensors. Preparing an entire facility for robot server swapping is a larger challenge. I don’t think we’re close to that. One thing data centers have going for them is that they are standardized. Robots are good at repeatable tasks with an unchanging canvas or environment. They can work with great precision in these type of environments.
A row of storage units housing Blu-Ray disks inside Facebook’s North Carolina data center. (Photo: Rich Miller)
That’s why robotic management is likely to be seen first at hyperscale operators who can control their entire environment, and who can begin to design their servers, racks and data halls with robotic management in mind. Amazon’s extensive use of robotics in its warehouses and distribution centers provides a glimpse of what type of optimization is possible when you design around the capabilities of robots. Unmanned or lightly-staffed environments also allow efficiency gains by allowing the facility to operate at higher temperatures and humidity, which has always been a focus for hyperscalers like Google and Microsoft.
In hyperscale data centers, you could design a rail-based system, much like the one created by TMGcore, with racks positioned adjacent to the rails. This would likely work best with racks positioned horizontally, rather than vertically, much as they are in the immersion tanks housing the servers at TMGcore.
An interesting wrinkle is that a robot-managed facility creates interesting possibilities for expanding vertically, stacking rows of data containers above one another. Human management and access would always be required, so platforms or gangways would need to be included.
A robot-managed facility creates interesting possibilities for expanding vertically, stacking rows of data containers above one another.
As we’ve noted on DCF, many data center developers are already building three and four-story buildings – or in the case of Facebook, a massive 11-story facility in Singapore – so this tracks with industry trends. This concept would pose some interesting challenges around cabling and design.
Colocation and multi-tenant data centers are a different story. These facilities have to be flexible to accommodate whatever diverse type of equipment their customers.
But one area where the light-out data center will become a priority is edge computing, which could soon see hundreds and perhaps even thousands of small data centers in distributed locations. As a practical matter, the majority of edge data modules and enclosures will need to operate without a human on site.
I believe that we are in the early days of robotics in the data center. While there have been interesting early projects, the greatest potential is yet to come.
