Amid a statewide power emergency, major data centers in Texas are running on generator power to keep customers online and reduce load on the state’s struggling grid. The state’s data center companies are also working to support their staff and communities amid an unprecedented winter weather emergency.
Digital Realty, Equinix and QTS Data Centers all reported that their data centers were providing uninterrupted service, with many facilities using emergency backup generators to maintain reliable power to customers and mission-critical agencies and networks.
Social media and news reports suggest that some data centers in the region are offline or experiencing problems. Some of these reports are coming from affected customers outside Texas, and given the power and communications issues in the state, it may take some time to fully assess data center uptime during this event.
Several of the publicly-held data center providers said they were managing through the outages.
“We’re keeping our people safe and keeping our customers engaged and getting through this together,” QTS President and CEO Chad Williams said Wednesday.
“Despite the extremely challenging weather conditions in Texas, Digital Realty has maintained operations for our data center customers in Dallas, Houston and Austin,” said Erich Sanchack, Executive Vice President for Operations at Digital Realty. “We have successfully transferred some loads to onsite generator power, while maintaining 100% uptime for critical and essential digital services as per our designs. By running generators at select sites, we have also been able to help reduce the overall power load on the ERCOT grid system.”
Sanchack said the company has offered one of its facilities in the Dallas region for use as a warming center for local residents, and is working with local charities and philanthropies to support critical needs. Digital Realty said it has also assisted customers in accessing diesel fuel during the crisis.
Stress Testing the Data Center Uptime Promise
The unusual winter weather has knocked 185 generation sources offline within ERCOT, the independent power grid serving most of Texas. As of Wednesday night, ERCOT said that more than 43,000 megawatts (MW) of generation have been forced off the system, including 26,500 MW of thermal generation (natural gas and oil) and nearly 17,000 MW of wind and solar.
Factors causing the generation outages include limited gas supplies, low gas pressure, frozen wind turbines and frozen instrumentation. Recovery has been slowed by the continued cold weather, but ERCOT said it was making better progress on Wednesday.
“We’re at a point in the restoration where we’re going to keep energizing circuits as fast as we safely can until we run out of available generation,” said ERCOT Senior Director of System Operations Dan Woodfin. That effort appeared to be making progress Thursday morning, when the number of customer outages fell below 1 million for the first time all week. It was unclear whether ERCOT would shift to managed rotating outages, as had been its initial plan.
Major disasters stress test of the primary value proposition of commercial data centers – keeping customers online, come what may. Extreme weather events and extended power outages document the industry’s ability to deliver uptime for customers in mission-critical circumstances, and also the occasions when data centers experience failures.
“This is why we have data centers,” said Raul Martynek, the CEO of DataBank.
Texas is a major data center destination, with nearly 200 commercial data centers and many more corporate facilities. Dallas/Fort Worth is one of the four largest data center markets in the country, and home to more than 3 million square foot of capacity.
While there have been no reports of the largest service providers or cloud platforms experiencing data center outages, social media reports suggest outages are occurring, and impacting customers and services.
- Florida-based Availity, which provides data services for more than 1 million health care providers and health plans, experienced an 11-hour outage Monday at its data center in Dallas. “In an effort to prevent interruption (from a utility outage), our data center provider attempted to transfer power from their primary system to their backup system,” Availity tweeted. “This transfer failed and caused an abrupt shutdown of our systems.”
- A data facility that supports government services in Austin was offline for much of the day on Monday, causing downtime for the city’s web sites as well as non-emergency phone systems.
- The California Medical Association, which provides insurance for low-income residents, said its website handling claims was offline due to problems with its Dallas-based data center.
- A credit union in Wisconsin said an outage at a Texas-based data center had knocked its mobile banking app offline.
For many Texas businesses, the loss of local power and water has made it hard to continue to operate. Companies with on-premises IT operations are struggling, especially when those facilities have limited backup power options. IT consultants told CRN that they are working with many customers to back up data and then manage a “graceful shutdown” of on-site
Extending A Hand Beyond the Data Center Walls
In extended regional power outages, the availability of diesel fuel for generators can become a major challenge, as was seen during Superstorm Sandy in 2012, when large parts of New York, New Jersey and Connecticut lost power for days. When every data center is running on generator power – along with hospitals and emergency operations centers and other life-critical customers – the system faces a stern test.
Sanchack said Digital Realty has not experienced issues procuring fuel. “By working closely with our supply chain partners, have been able to keep our generators fully fueled,” said Sanchack. “Additionally, we have provided support by redirecting fuel supplies to other customers in Texas who have data center capacity outside the Digital Realty portfolio of properties.
“From a community outreach perspective, Digital Realty has also offered up the use of one of our data center spaces in Lewisville, TX to provide potential overflow facilities to the city for families who lack access to power and heat,” he added. “This response is being coordinated with the city administration as a backup option to help those in need. We are also looking to make charitable contributions to local philanthropic organizations in Texas also on the frontlines, as part of our overall approach.”
- CyrusOne, which has 12 data centers spread across the Houston, Dallas, Austin and San Antonio markets, reports that all of its facilities have been “stable.” “We’ve maintained 100 percent uptime, and just some minor cooling issues,” said John Hatem, Chief Operating Officer at CyrusOne. “This is an unprecedented situation, and what our teams on the ground have been doing is just amazing. It’s been an all-out effort by the teams to keep our customers running, resulting in no major impacts.”
- Equinix, which operates nine data centers in Texas, said “all sites are operational, running on utility power where available and backup generators where utility power is not available.” Equinix did not comment on specific facilities, but customer reports from mailing lists indicate the Dallas Infomart, the primary carrier hotel and network hub in the Dallas market, went on generator power early Monday, with the company saying “the (Infomart) IBX will remain on generator power until utility power is stable.”
- DataBank said all eight of its Texas facilities in Dallas, Austin and Waco are operating without interruption. “DataBank is performing its mission of keeping our customers’ IT infrastructure powered and cooled so that their data is protected,” said Martynek. ” We topped off our fuel before the storm and have arranged for deliveries in advance.” The company said it is monitoring fuel use in real-time, diverting deliveries based on usage, and sourcing backup fuel from out-of-state providers. As a contingency, DataBank has equipped all Texas facilities with sufficient food and water supply and provided nearby hotel accommodations for on-site staff, if next-shift personnel are unable to travel safely to the data centers.
- Microsoft Azure has one of its major cloud hubs in San Antonio. “Microsoft is closely monitoring the situation in Texas. Our Datacenter teams are in continuous contact with local power companies and ISPs to ensure we have the most up-to-date information,” the company said.
- Evoque said its two Dallas-area facilities remained online. “We have had no loss of power, and our clients have had consistent access to their corporate-critical data, applications and workloads,” said Drew Leonard, VP for Strategy at Evoque. “In addition, as a commitment to the communities we serve, we chose to power our Allen data center by generator for 15 hours, returning multiple megawatts of power back to the grid. Our clients saw no interruption in uptime during that time.”
A Green Benefit to Power Contracts
Consumers in Texas have complained of large price hikes when power is available. On Monday evening the Texas Public Utility Commission (PUC) directed ERCOT to maintain higher energy prices to reflect the growing scarcity of power, saying “pricing mechanisms were not generating an optimal response” to conditions.
QTS President and CEO Chad Williams said the company’s use of renewable energy for its data centers helped insulate customers from pricing risk. In 2019 QTS arranged 10-year contracts for wind power to support its data centers in Irving and Fort Worth.
“We’ve had a long-term strategy in sustainability in the way we purchase power,” Williams said Wednesday on the company’s quarterly earnings call. “The commitment to renewable energy comes with stable pricing because of our long-term contracts.”
QTS CFO Jeff Berson said the contracts have fixed pricing, and QTS is “not anticipating any material impact for customers or for QTS.”
CyrusOne said uses hedging strategies to manage power pricing in the Dallas and Houston markets, and said the company is “in good shape” on pricing risk.
Texas is unique in that it is decoupled from the interconnected power grids serving the eastern and western United States. Among the contiguous 48 states, Texas is the only one that has a standalone electric grid entirely within the state. This means that with few exceptions, Texas produces the electricity consumed within its borders.
For many customers, this has actually been a selling point for data center site selection, as Texas was perceived as protected from the cascading outages in which problems on interconnected grids ripple across regions, as in the 2003 Northeast blackout. The current grid outages will add a new wrinkle to consider.
