AT&T is working to repair key infrastructure after a Christmas day explosion damaged its regional telecom hub in Nashville, disrupting telecom services across much of the Southeast. Police say the “intentional” detonation of a recreational vehicle killed the bomber, but have not yet determined a motive for the incident.
The blast knocked out the primary power connections for the AT&T central office at 185 Second Avenue, and caused flooding that knocked out the building’s emergency backup generators. The resulting outage interrupted 911 services in Tennessee and Kentucky, and caused mobile network outages across the region.
The network issues impacted air traffic control system at Nashville International Airport, prompting the FAA to halt flights for several hours on Christmas Day. Many businesses in the Nashville reported they were temporarily unable to process credit card transactions.
AT&T said it expected to restore all services by Sunday evening. The company was using portable generators to restore power to upper floors of its damaged building, and deployed 25 temporary satellite cell towers and 24 trailers of disaster recovery equipment across the Nashville region to help restore mobile phone service.
As of Sunday afternoon, AT&T said 96% of its wireless network had been restored, along with 60% of business services are restored, and 86% of consumer broadband and entertainment services.
On one level, the extensive restoration of network services within 72 hours demonstrated the resilience of communications infrastructure, especially given the extensive physical damage to a key network hub.
But the outage also prompted a discussion on network news shows about the vulnerability of telecom and data infrastructure to attacks, with some officials and analysts expressing concern that an individual attacker could disrupt regional communications, including 911 systems.
No Conclusions Yet on Bomber’s Motive
Authorities have not revealed any findings about the motive of the bomber, Anthony Q. Warner, and whether the AT&T facility was his intended target. Reuters and other media reported that Warner’s late father worked at BellSouth, suggesting the possibility that Warner may have been familiar with the building or had a specific interest in the company.
The explosion, which was captured on a Nashville police camera, damaged 40 businesses and caused one building to collapse. Details of the incident suggested the goal may have been property damage, given the timing early on Christmas morning and a recording blaring from the RV warning people to evacuate because an explosion was imminent. Nashville police were able to warn and evacuate residents. There were no deaths, and three people were injured.
The AT&T facility “suffered significant damage in the blast,” said Jeff McElfresh, CEO of AT&T Communications. “That facility includes connection points for regional internet services as well as local wireless, internet and video.”
Free Resource from Data Center Frontier White Paper Library
Aligned Enhanced Soft Water and Water Management Saves Big at Phoenix Data Center
It takes a tremendous amount of energy and often copious amounts of water to power and cool any data center. In fact, data centers contribute to industry’s consumption of 45% of all available clean water. Get the new report from Aligned Energy that highlights how the company ramped up soft water and water management to save a total of 24 million gallons of water annually at its Phoenix data center.Get this PDF emailed to you.
Structural Damage and Flooding Pose Challenges
The AT&T building on Second Avenue is a central office (CO), a telecom exchange where physical lines converge and switching equipment routes calls and data between customers. There are about 25,000 central offices in the U.S., which manage local calls in a particular area. Regional hubs like the AT&T building coordinate traffic between other COs and exchanges.
“In the hours that followed the explosion, our local service remained intact through temporary battery power,” said McElfresh. “Unfortunately, a combination of the explosion and resulting water and fire damage took out a number of backup power generators intended to provide power to the batteries. That led to service disruptions across parts of Tennessee, Kentucky and Alabama. More than 48 hours later, some customers are still experiencing outages.”
The company said commercial power connections are “damaged and offline” and it is working with the local power utility on repairs. The basement was flooded with more than three feet of water, which was pumped out early Saturday. AT&T then began to restore power to parts of the building using portable generators, punching holes through the walls (working closely with safety and structure engineers) to run cables and connect equipment to the portable power units. The company said it is also working on the cooling equipment to manage the temperatures in the facility.
AT&T said the blast also damaged the elevators, some beams and columns, and the building’s façade.
“What has made network restoration so difficult is doing it while maintaining the integrity of an active crime scene in cooperation with federal and local law enforcement,” said McElfresh. “Hundreds of employees – our own AT&T employees as well as first responders – have stepped in over the last two days to restore service.
Addressing Nashville residents, McElfresh said the explosion “shook your peace of mind and scarred the community that you call home. AT&T is a part of that community. We live here. We do business here. And we’re in this with you. We know it is frustrating and we apologize for the inconvenience.”
Data Center Frontier is edited by Rich Miller, the data center industry’s most experienced journalist. For more than 15 years, Rich has profiled the key role played by data centers in the Internet revolution.
ATT had too much equipment in one building. As far as the back-up batteries are concerned
backup batteries are designed to last 24 hours. The backup batteries are typically
charged by La Marche charging machines. Generator to charge one set of batteries by
La Marche charger while another set of batteries runs the equipment. The generator is
powered by natural gas not by the local power company. The preferred location of
the generator is never in the basement of the building but on the third floor away
from storms or hurricanes.