At an electric station near Phoenix, BASELAYER and Salt River Project are testing a new way for utilities and data centers to work together. With the DataStation project, the two companies hope to shift hyperscale cloud farms closer to the heart of the power grid, an architecture that promises major benefits for both industries.
“Data centers traditionally have been picking up power on the bottom end of the supply chain, which was designed for residential and industrial customers,” said Clint Poole, manager of SRP Telecom, a unit of Salt River Project. “It’s the difference between freeways and neighborhood roads. The freeways and bulk transmission system have far more capacity.”
The DataStation project connects the data center to the fast lane of the largest freeways on the power grid, the bulk transmission lines and substations. Moving nearer to where power is generated will make it easier for data centers to provision large amounts of electricity, and to gradually expand their capacity over time. The bulk transmission system also offers improved reliability, which could allow cloud builders to operate without expensive backup equipment.
No UPS, No Generators
“The most reliable point on the grid is the bulk transmission receiving station,” said Poole. “We have receiving stations with 100 percent reliability from the time they were built.” To prove its point, the “proof of concept” DataStation being built by SRP and BASELAYER will operate without any diesel generators, gathering data to document the reliability of this approach.
The BASELAYER modular data centers are built in a factory near Phoenix. (Photo: BASELAYER)
That’s not a step every data center operator is willing to try. Until fairly recently, this type of experiment would not have been feasible. The key enabler, Poole says, is the modular data center design from BASELAYER, a technology company spun off from Phoenix-based IO. These data halls are built in factories and come ready to deploy on site, often resulting in a cheaper and faster construction process. Best of all, rather than extending the power to the data center, modular designs allow the IT workloads to move closer to the power source.
“The module is really what started this conversation,” said Poole. “The modular data center is disruptive and innovative. It’s true disruption is when you move it to where it has the greatest impact.”
Moving Up the Power Supply Chain
Salt River Project (SRP) provides electricity to more than 1 million retail power customers in a 2,900 square mile territory spanning the Phoenix market. It’s an integrated utility, meaning that it serves the entire supply chain, including power generation, transmission, distribution, and metering and billing.
Phoenix has been a growth market for data centers over the past decade, including major new projects for eBay, IO, Digital Realty, CyrusOne, Phoenix NAP and NextFort, along others. Demand for power is beginning to outpace the supply in some parts of its infrastructure. That includes Chandler, home to several major data centers, where SRP projects it will see capacity constraints by 2017. The utility is planning to expand its transmission system, adding 230 kV overhead lines and two new substations to supplement the existing 69 kV transmission system.
This is a scenario that is likely to be repeated as companies shift their IT workloads from in-house facilities to cloud platforms and colocation facilities. The IT outsourcing trend is not new, but has been accelerated by corporate belt-tightening and the emergence of cloud computing models that simplify the process. That’s driving the construction of more new data centers, including large server farms that require enormous amounts of electric power.
Averting Capacity Constraints
“From a power perspective, we need to think about how the data center industry at some point will grow beyond the utility industry’s ability to support it,” said Poole, noting projections that data center energy use will continue to grow. Data centers used 91 billion kilowatt hours of electricity in 2013, according to the Natural Resources Defense Council, which projects that will rise to 138 billion kwH by 2020. The largest cloud providers are growing even faster, as Amazon Web Services saw its revenue rise 49 percent over the last 12 months.
“We could accommodate that type of increase today if it was located further up our grid,” said Poole. “The bulk transmission system was built to be highly scalable.”
The data center industry is adding power demand at the edge of the grid, which is also the most difficult place to expand the infrastructure. Poole believes that moving “up the supply chain” and siting data centers near bulk transmission lines provides clear benefits to both utilities and data center operators.
An illustration of the DataStation locations, compared to the traditional location of data centers. (Source: Salt River Project)
According to SRP, shifting data centers to the bulk transmission sector of the grid offers a number of advantages:
- Scalability: It’s far easier and cheaper for utilities to add capacity at the bulk transmission level than in industrial parks and residential areas. Shifting demand up the chain eases load on the grid. It also could make it easier for data center providers to procure large up-front power capacity, or to expand existing capacity. “One way to accommodate that right now is to have the largest building – the most-stranded asset,” said Poole. “We’re offering the same opportunity without stranding the asset. You can grow at your own pace.”
- Connectivity: Shifting capacity to bulk transmission stations could boost usage of fiber networks operated by utilities, including SRP, whose network connects with many of the large data centers in its service area. “Everywhere we have transmission, we also have dark fiber,” said Poole. “I’m sure other utilities do as well.”
- Reliability: Poole says SRP’s Extra High Voltage bulk transmission lines have reliability of seven nines – 99.99999 percent uptime – which exceeds the five and six nines track records of top-tier data centers supported by UPS units and emergency backup generators. With current designs “you have to build reliability into the data center itself,” using generators, UPS units and batteries, said Poole. “Where you place a data center on the grid DOES matter. We need to move data centers to the most reliable parts of the grid.”
- Business Models: Deploying data centers modules at bulk transmission sites effectively allows utilities to build businesses around real estate development and leasing wholesale data center space.
“It’s the difference between being pro-active and reactive,” said Poole. “This gives the utility industry the opportunity to be attractive and address this growth in demand. I think all utilities, not just SRP, should be looking at this.”
Bringing data centers to the grid’s “freeway” also averts the need to boost capacity through expansion projects that can are always expensive and can sometimes be controversial.
Salt River Project has temporarily suspended its plans for the 230 kV transmission line to support future growth in Chandler, citing challenges in securing rights of way from area land owners. A group of residents have opposed the plan, objecting to overhead power transmission towers and urging SRP to bury the lines, which the utility says would be 10 times as expensive. SRP says it will continue to pursue alternative routes to complete the project.
A similar dynamic has emerged in Prince William County in northern Virginia, where Dominion Power is seeking to build a 230 kV transmission line to support a proposed Amazon Web Services data center in the town of Haymarket. A group of residents has formed The Coalition to Protect Prince William County, which opposed the overhead transmission lines and advanced an alternate plan for buried lines along an alternate route.
Embracing the Modular Advantage
What no one knew at the time was that Google had already built an entire data center of containers packed with servers. Microsoft would soon follow suit. These first-generation containers were also used to run data centers in remote or harsh environments. Soon hardware vendors like Dell, HP, Rackable and SGI were offering containerized data centers, preferably to be filled with their own servers and storage.
But in 2010, Phoenix-based colocation provider IO and its CEO, George Slessman, introduced a different vision for a modular product: the enterprise IT market. The IO.Anywhere module could be deployed at one of IO’s massive data halls, or at a parking lot outside a corporate office. IO also looked at the data module as part of a larger system of components that could be refined and customized, an approach also being advanced overseas by BladeRoom, Colt and DataPod. IO then went a step further and created software to manage a customer’s entire infrastructure.
The Anywhere Enterprise
Slessman made clear that this wasn’t simply another form factor to offer customers with specialized needs. Rather, the Anywhere initiative was a bold move to place modular designs at the heart of its pitch to enterprise customers, and create factory-built environments that looked just like the traditional enterprise data centers. By late 2014, that strategy had paid off in a list of marquee customers including Goldman Sachs, LexisNexis, Allianz, CBS and CenturyLink.
In December 2014, IO spun off BASELAYER as a technology vendor, selling modules and data center management software, while IO retained the colocation business and operated the data center facilities. One of BASELAYER’s first jobs is using its modules to help Salt River Project realize its vision, which fits neatly with BaseLayer’s prime directive.
“The partnership with SRP is a perfect example of how BASELAYER develops technology to challenge the status quo,” said BASELAYER CEO William Slessman. “We’re bringing like-minded people together across industries to build a better future.”
The key building block for the DataStation project is the BASELAYER Edge Data Module, a 42-foot long enclosure that can house up to 20 racks of server equipment. The 13-foot-high module uses extra-tall racks, each measuring 52RU (“rack units”), providing up to 500 kilowatts of electricity for the IT equipment. The modules each weigh about 57,000 pounds – nearly 30 tons – but can travel via truck, rail or ship.
“Our design philosophy is ‘the right size in the right place at the right time,'” said Samir Shah, Vice President for Product Management and Marketing at BASELAYER.
On the utility side, SRP is creating a docking station that allows the DataStation to connect to an adjacent multi-sourced 230kV station, and also ties into a 69 kV station. “We have several locations on our grid where our bulk transmission lines can work,” said Poole, who noted that the utility can certify the module as a piece of equipment, rather than a building.
Poole says the DataStation’s advantages are not limited to reliability and scalability. The project also offers potential savings by eliminating equipment from the power distribution chain.
“You’re taking out your transfer switches, UPSes and on-site generators,” he said. “The data center industry has re-invented the grid within their own facility. All of this is baked into the grid.”
The project is energized and nearing completion, and scheduled to go live in August.
Leading With the Cloud
Poole believes the proof of concept is the first step on a revolution in data center site selection.
“We’re fully committed to commercializing the DataStation,” said Poole. “This (proof of concept) represents SRP’s commitment to bring this technology to market. We’re building this, and we’ll be testing and quantifying this design. This could be something much bigger.”
If all goes well, Poole says the POC will lay the groundwork for a commercial data center at one of the company’s bulk transmission stations, using BASELAYER modules as wholesale data center space. Since the data halls can be delivered on demand, the project can be expanded in phases. Poole says the commercial site could offer up to 40 megawatts of IT capacity.
Will data center customers buy into the concept? It won’t be for everybody, but Peter McNamara of BASELAYER says a growing number of customers are open to new deployment models.
“The market drivers have changed,” said McNamara, the EVP of Sales and Customer Operations. “The traditionalists are being forced to look at data centers in different ways. They’re being forced to examine why they’re building their own data centers with 30 to 40 percent capacity utilization. Acceptance of modular has come around, and there’s been a tremendous shift to outsourcing in the cloud.”
Some of the early adopters will be cloud providers with large footprints, said Shah. “I see a huge advantage for high-density cloud players to be the tip of the spear.”
The Road Ahead
Poole, for one, believes the economics will get customers’ attention. “I think it comes down to TCO,” said Poole. “Price seems to be the number one consideration in the industry right now. The cost element of this will be disruptive. It will be substantially less (than current pricing).
“I truly see the DataStation as an industry-changing innovation,” he said. “We believe SRP is uniquely positioned to deliver this solution.”
BASELAYER’s McNamara believes the acceptance of modular has come around, with many more companies becoming comfortable with factory-built data centers, persuaded by the opportunity for a faster and cheaper deployment process.
“The momentum is in our favor,” said McNamara. “The POC is a tremendous opportunity for us. It’s the logical next step in delivering data center capacity for end users. You can see the convergence of power, network, water and land. It’s all about how to distribute content and data to end users in the most efficient way. We believe modular is the way to go.”
