Scaling Up the Lights Out Data Center

The “lights out” server farm has been living large in the imaginations of data center futurists. It’s been 10 years since HP first made headlines with its vision of unmanned data centers, filled with computers that monitor and manage themselves. Even Dilbert has had sport with the notion.

But the list of those who have successfully implemented lights out data centers is much shorter. HP still has humans staffing its consolidated data centers, although it has used automation to expand their reach (each HP admin now manages 200 servers, compared to an initial 1-to-15 ratio). In 2011, AOL announced that it had implemented a small unmanned data center, but that doesn’t appear to have progressed beyond a pilot project.

EdgeConneX is changing that. The company has pursued a lights out operations model in building out its network of 24 data centers across the United States and Europe. EdgeConneX, which specializes in content distribution in second-tier markets, designs its facilities to operate without full-time staff on site, using sophisticated monitoring and remote hands when on-site service is needed.

The EdgeConneX design is perhaps the most ambitious example yet of the use of automation to streamline data center operations, and using design as a tool to alter the economics of a business model.

The Deployment Template as Secret Sauce

The key to this approach is an advanced design and operations template that allows EdgeConneX to rapidly retrofit existing buildings into data centers with Tier III redundancy that can support high-density workloads of more than 20kW per cabinet. This allowed the company to deploy 18 new data centers in 2014.

A lean operations model was baked into the equation from the beginning, according to Phill Lawson-Shanks, the Chief Solution Architect at EdgeConneX.

“Our primary build is a 2 to 4 megawatt data center and about 10,000 square feet,” said Lawson-Shanks. ” We always build with a view that we’ll have to expand. We always have an anchor tenant before we go to market.”

That anchor is usually a cable multi-system operator (MSO) like Comcast or Liberty Global, which are both investors in EdgeConneX. Comcast was just announced as the anchor in the new EdgeConneX facility in Jacksonville, Fla., and has a presence in many EdgeConneX facilities.

Solving the Netflix Dilemma

“We’re helping the cable companies solve a problem: to get Netflix and YouTube off their backbones,” said Lawson-Shanks. “The network is being overwhelmed with content, especially rich media. The edge is growing faster than you can possibly imagine.”

We’ve previously covered the explosion of content caching at the edge of the Internet (Data Centers on the Edge: Streaming and IoT Reshape the Network) as well as the need for matching the size of edge facilities to the level of demand (Right-Sizing for the Edge: Data Center Design in Secondary Market).

Data center site selection is extremely important in the EdgeConneX model. In each new market, the company does extensive research of local network and telecom infrastructure, seeking to identify an existing building that can support its deployment template. Once it selects a property and closes the transaction, EdgeConneX commences construction, installing its standard infrastructure platform of  mechanical and electrical equipment.

“This is a patented operations management system and pricing model that makes every Edge Data Center a consistent experience for our customers nationwide,” Clint Heiden, Chief Commerical Officer of EdgeConneX, told Telecom Ramblings. “If we walked you through our Pittsburgh site and then put you in our Salt Lake City facility, you wouldn’t notice a difference.”

Managing Infrastructure from Afar

The lynchpin of the lights out approach is data center infrastructure management (DCIM) doftware. EdgeConneX uses a patented data center operating system called EdgeOS to monitor and manage its facilities. The company has the ability to remotely control the generators and UPS systems at each data center.

EdgeConneX facilities are managed from a central network operations center in Santa Clara, with backup provided by INOC, which specializes in remote network operations. When on-site maintenance is required, remote hands support is provided by Salute Mission Critical, which provides on-demand technicians in all 50 states. Salute specializes in hiring veterans of the U.S. military.

Currently 20 of the 24 EdgeConnex data centers are unmanned. Each facility has a multi-stage security system that uses biometrics, PIN and keycard access, with secured corridors (“mantraps”) and video surveillance. The company provides manned security and IT staffing in situations as needed in the remaining data centers.

Many EdgeConneX customers deploy high-density installations using blade servers or equipment like Cisco UCS (Unified Computing System), which is why the facilities are designed to support workloads of 20kW per rack, and sometimes as much as 30kW a rack. EdgeConneX doesn’t currently use aisle containment in its design, but Lawson-Hicks said the company is studying whether to include it in future designs.

The Road Ahead

EdgeConneX expects to be building data centers for some time to come. Demand for edge-based content caching is growing fast, and the company is also starting to see demand from cloud providers seeking distributed space.

“The growth for us is people who want to be close to the on-ramps,” said Lawson-Shanks, who believes the need for edge-based content will be driven by changes in both technology and demographics.

“The Internet of Things is going to be huge for us,” said Lawson-Shanks, referring to the surge in data collection from networks of sensors in both consumer devices and industrial infrastructure. “There will be this hub-and-spoke (distribution of content), but I think there is a bigger shift coming as millennials build and use new apps.

“The user experiences and devices are changing,” he said. “But fundamentally, it’s latency, latency, latency.”

Much of this technology wasn’t in the mix in 2005 when the first visions emerged of an unmanned data center. But as we see edge data centers proliferate, the EdgeConneX model has demonstrated the possibility of using automation to approach these facilities differently. This approach won’t be appropriate for many types of workloads, as most data centers in second-tier and third-tier markets will serve local businesses with compliance mandates that require high-touch service from trained staff.

But one thing is certain: The unmanned “lights out” data center is no longer a science project or flight of fancy. In 20 cities across America, it’s delivering Netflix and YouTube videos to your devices.