An emergency generator caught fire at a data center in Ogden, Utah over the Easter weekend, causing the full shutdown of the data center and lengthy outages for customers. The incident at a WebNX facility comes a month after a more serious fire that destroyed an entire OVH data center in Strasbourg, France.
Dedicated server hosting firm WebNX says a backup generator experienced a “catastrophic failure” Sunday afternoon, and fire officials opted to cut power to the entire building. The company says a small number of servers may have suffered water damage as emergency crews battled the generator fire.
“Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on,” the company said. “However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol.
“Customers servers in one of our main bays were exposed to water and possible damage may have occurred,” WebNX added. “No fire damage was inflicted on customer servers. The majority of hardware in the entirety of our data center was spared, but there are machines that need to be inspected for water damage, and possibly rebuilt.”
WebNX has been providing regular customer updates through its Facebook page and web site and said it is still working on restoring full power and restarting customer servers. As of Wednesday afternoon, the company said many servers were back online, and that they were working through issues on remaining servers that were damaged in the incident.
“The majority of hardware in the entirety of our data center was spared, but there are machines that need to be inspected for water damage, and possibly rebuilt,” WebNX said in an update Wednesday. “Timelines are difficult to predict at any stage and many clients’ servers won’t be back online for several weeks.”
“If you are unable to access your server, it can be assumed it is currently down and will remain so until it can be inspected for any damage in the coming weeks. We are optimistic that the vast majority of data stored on these servers is recoverable. Several servers racked in the vicinity of the incident have resumed operation without trouble.”
The company’s roots date to the early Internet BBS days, and it launched its web hosting operation in 1999. The company operates data centers in Downtown Los Angeles and in Ogden, where it has a 100,000 square foot data center.
Fire Risk and Lessons for the Data Center Industry
The incident, which was first reported by The Register, comes a month after an OVH data center was destroyed by fire, with a second data center on the campus ultimately being shuttered due to damage from the fire.
The March 9 event in France also has caused extended downtime for customers, as fire officials turned off for the entire four-building campus. Once the site was secured, OVH began a cleanup effort with plans to restore service at the remaining three data centers.
Free Resource from Data Center Frontier White Paper Library
Far Enough to be Safe, Close Enough to be Fast: A Field Guide for Adding Geographic Diversity to Your Disaster Recovery Plan
Mission-critical IT operations can be impacted by disasters that are both naturally occurring, such as pandemics, hurricanes, earthquakes, tornados, floods or wildfires – as well as those that are man-made, including cyber attacks, terrorism or the inadvertent failure of infrastructure systems like power grids. Get the new report from Databank that highlights how physical distance is one of the first considerations in designing or selecting a disaster recovery solution.Get this PDF emailed to you.
Fires that consume an entire data center building are rare due to the industry’s standard use of data center fire suppression technology, which is designed to detect smoke and then deploy suppressants (either water or chemical agents) to extinguish the fire and contain the damage to a small area.
But the WebNX event will likely intensify the data center industry’s efforts to re-evaluate assumptions about fire protection in the wake of the fire at OVH. On this front, two recent columns are worth noting:
- In Learning From the OVHcloud Data Center Fire, Andy Lawrence of The Uptime Institute says there have been 25 data center fires in recent industry history, including 11 which have not been publicly reported. Lawrence reviews the known details from the OVH event and highlights some design and operational issues for other data center providers to consider.
- In Give OVH a Break and Use the Data Center Fire as A Teachable Moment, Dennis Cronin of DCIN (the Data Center Incident Reporting Network) writes that the OVH incident “should not be assumed unique to OVH. Everyone in the global data center industry has something to learn from this disaster and probably many things to learn. This includes clients, support vendors, designers, contractors, and staff. It will be a tragedy if the facts and the lessons learned are restricted to a small exclusive group and not shared with the data center industry.”
Data Center Frontier is edited by Rich Miller, the data center industry’s most experienced journalist. For more than 20 years, Rich has profiled the key role played by data centers in the Internet revolution.
Great read Rich thank you
There’s really no excuse to have any type of catastrophic incident lead to customers having to deal with DAYS of lost service. It’s totally unacceptable for any provider. What blows my mind is that in this day and age the center had to deal with WATER damage… really… WATER DAMAGE… why is water ever used in an electrical fire incident in the first place? Just ridiculous.
Stops combustion pretty fast.
Don’t be a dope. Why would a data center responsible for millions of dollars of equipment be housed in a combustible space? If you’re going to be a player in this space, my view is that you house the equipment in a non-combusitble building where you can tackle an electrical fire WITHOUT WATER, period. It is the 21st century, figure it out.
Plenty of excuses I can think of. generators fail often. Patience Don. No need to say dope. Happy days.
Apologies. Hopefully hard lessons will be learned and actions taken to circumvent future events causing catastrophic consequences.