An emergency generator caught fire at a data center in Ogden, Utah over the Easter weekend, causing the full shutdown of the data center and lengthy outages for customers. The incident at a WebNX facility comes a month after a more serious fire that destroyed an entire OVH data center in Strasbourg, France.
Dedicated server hosting firm WebNX says a backup generator experienced a “catastrophic failure” Sunday afternoon, and fire officials opted to cut power to the entire building. The company says a small number of servers may have suffered water damage as emergency crews battled the generator fire.
“Sunday afternoon the city power was disrupted and, as designed, our backup generators automatically switched on,” the company said. “However, during that transition, one of our backup generators that had been recently tested and benchmarked specifically for this situation experienced a catastrophic failure, caught fire, and as a result initiated the fire suppression protocol.
“Customers servers in one of our main bays were exposed to water and possible damage may have occurred,” WebNX added. “No fire damage was inflicted on customer servers. The majority of hardware in the entirety of our data center was spared, but there are machines that need to be inspected for water damage, and possibly rebuilt.”
WebNX has been providing regular customer updates through its Facebook page and web site and said it is still working on restoring full power and restarting customer servers. As of Wednesday afternoon, the company said many servers were back online, and that they were working through issues on remaining servers that were damaged in the incident.
“The majority of hardware in the entirety of our data center was spared, but there are machines that need to be inspected for water damage, and possibly rebuilt,” WebNX said in an update Wednesday. “Timelines are difficult to predict at any stage and many clients’ servers won’t be back online for several weeks.”
“If you are unable to access your server, it can be assumed it is currently down and will remain so until it can be inspected for any damage in the coming weeks. We are optimistic that the vast majority of data stored on these servers is recoverable. Several servers racked in the vicinity of the incident have resumed operation without trouble.”
The company’s roots date to the early Internet BBS days, and it launched its web hosting operation in 1999. The company operates data centers in Downtown Los Angeles and in Ogden, where it has a 100,000 square foot data center.
Fire Risk and Lessons for the Data Center Industry
The incident, which was first reported by The Register, comes a month after an OVH data center was destroyed by fire, with a second data center on the campus ultimately being shuttered due to damage from the fire.
The March 9 event in France also has caused extended downtime for customers, as fire officials turned off for the entire four-building campus. Once the site was secured, OVH began a cleanup effort with plans to restore service at the remaining three data centers.
Fires that consume an entire data center building are rare due to the industry’s standard use of data center fire suppression technology, which is designed to detect smoke and then deploy suppressants (either water or chemical agents) to extinguish the fire and contain the damage to a small area.
But the WebNX event will likely intensify the data center industry’s efforts to re-evaluate assumptions about fire protection in the wake of the fire at OVH. On this front, two recent columns are worth noting:
- In Learning From the OVHcloud Data Center Fire, Andy Lawrence of The Uptime Institute says there have been 25 data center fires in recent industry history, including 11 which have not been publicly reported. Lawrence reviews the known details from the OVH event and highlights some design and operational issues for other data center providers to consider.
- In Give OVH a Break and Use the Data Center Fire as A Teachable Moment, Dennis Cronin of DCIN (the Data Center Incident Reporting Network) writes that the OVH incident “should not be assumed unique to OVH. Everyone in the global data center industry has something to learn from this disaster and probably many things to learn. This includes clients, support vendors, designers, contractors, and staff. It will be a tragedy if the facts and the lessons learned are restricted to a small exclusive group and not shared with the data center industry.”
