• About Us
  • Partnership Opportunities
  • Privacy Policy

Data Center Frontier

Charting the future of data centers and cloud computing.

  • Cloud
    • Hyperscale
  • Colo
    • Site Selection
    • Interconnection
  • Energy
    • Sustainability
  • Cooling
  • Technology
    • Internet of Things
    • AI & Machine Learning
    • Edge Computing
    • Virtual Reality
    • Autonomous Cars
    • 5G Wireless
    • Satellites
  • Design
    • Servers
    • Storage
    • Network
  • Voices
  • Podcast
  • White Papers
  • Resources
    • COVID-19
    • Events
    • Newsletter
    • Companies
    • Data Center 101
  • Jobs
You are here: Home / Energy / Facebook: Open Sharing Was Key to Addressing Arc Flash Incidents

Facebook: Open Sharing Was Key to Addressing Arc Flash Incidents

By Rich Miller - October 19, 2017

Facebook: Open Sharing Was Key to Addressing Arc Flash Incidents

An overhead busway at the Facebook Lulea data center after an arc flash incident in 2014. The company presented about its incident response during this week's 7x24 Exchange Conference. (Image: Facebook)

LinkedinTwitterFacebookSubscribe
Mail

PHOENIX, Ariz. – Data center operators don’t love talking publicly about failure. Ranking high on the list of those unwelcome failures is arc flash.

That’s why it was notable when executives from Facebook and Schneider Electric took the stage at this week’s 7×24 Exchange Fall Conference to discuss two arc flash incidents at Facebook’s data center in Sweden, and how the company’s post-event analysis and response headed off other potential incidents.

An arc flash is an electrical explosion that generates intense heat that can reach 35,000 degrees F, which can damage and even melt electrical equipment. Arc flash incidents also represent a significant threat to worker safety.

“How we talk about failure is important,” said James Swensen, Senior Global Facilities Operations Manager at Facebook. “Both of our organizations were concerned about sharing this story. No one wanted us to show a picture.

“It’s uncomfortable,” he continued. “It’s not fun. But if you don’t go through that pain, you’re not going to get to the lessons learned.”

Arc Flash on Overhead Busway

The arc flash incidents occurred in 2014 and 2015 during live operation of Facebook’s first international data center in Lulea, Sweden. No one was injured in either incident, and since the events were limited to one of the facility’s four power rooms, the data center never went offline.

Even so, an arc flash is a dramatic event in a data center, where the power infrastructure handles enormous amounts of electricity. Each data hall within a Facebook facility houses tens of thousands of servers. As a result, the potential cost of mistakes is high.

The incidents in Lulea occurred in an overhead busway, an enclosure housing copper bars to conduct electricity, which distributes power within the data center. The 5,000 amp busway was assembled in sections that are each 10 feet long and weigh 700 pounds. These sections must be raised and pieced together and joined to the busway.

The complexity of the assembly was heightened by large number of parties involved in the install, according to Schneider Electric, which didn’t participate in the initial assembly but was brought in for the post-event analysis. “This isn’t a typical data center busway design,” said Bill Westbrock, Global Account Manager with Schneider Electric. “It’s a lot of power in a small space.”

At Issue: Torquing of Busway Joints

The analysis focused on the busway joints, where the sections came together. Schneider’s analysis found that the bolts that secure the joint were not torqued (tightened) properly during assembly. Over time, this created a “thermal avalanche” effect that eventually led the joint to fail. Here’s a diagram from the presentation:

Free Resource from Data Center Frontier White Paper Library

reliability-as-a-servce
The Affordable Microgrid: Securing Electric Reliability through Outsourcing
Microgrids, which use controllers to connect multiple power generation and storage sources, can provide electric reliability but they can also be too complex and costly for businesses to tackle on their own. Enchanted Rock explains how the reliability-as-a-service model makes microgrids affordable while providing electric reliability.
We always respect your privacy and we never sell or rent our list to third parties. By downloading this White Paper you are agreeing to our terms of service. You can opt out at any time.

Get this PDF emailed to you.

A diagram of how inadequate tightening of a bolt in a busway joint led to an arc flash. (Image: Facebook, Schneider Electric)

A diagram of how inadequate tightening of a bolt in a busway joint led to an arc flash. (Image: Facebook, Schneider Electric)

The analysis enabled two key actions. The first was a review of busway joints across the Lulea facility and all other Facebook data centers using this design.

“In a matter of hours and days, people in other buildings with similar busways were checking for the issues that we had found,” said Swensen.

The review revealed three other locations in Facebook facilities where the joint bolts were not properly torqued. Swensen says he is convinced this averted additional arc flash incidents.

Schneider’s root cause analysis found “many potential failure points” in a detailed analysis of how a joint bolt wound up not being properly torqued. Among the issues raised in the review was the inadequacy of periodic thermal scans designed to identify “hot spots” from emerging problems in busway joints.

In response, Schneider designed temperature sensors that could be deployed along the busway to provide real-time monitoring of temperature along the busway joints and issue alerts. Here’s an oveview:

Addressing Arc Flash in Data Centers

Arc flashes have led to expensive outages in data centers, most notably a 2009 incident at Fisher Plaza in Seattle, which knocked a major online payment gateway offline, slowing global e-commerce. The event was blamed on an insulation failure on a busway, and the building owner incurred $6.8 million in cash expenses related to the fire, including remediation and capital projects.

There were a series of arc flash incidents during the construction and testing of the NSA data center in Utah, according to media reports, which said the events caused significant equipment damage.

The larger danger with arc flash is worker safety. According to the American Society of Safety Engineers, more than 3,600 workers suffer disabling electrical contact injuries annually.

Reducing arc flash hazards has been a growing priority for data center power vendors and the National Fire Protection Association (NFPA), which in 2012 introduced new regulations designed to limit the scenarios in which technicians are working with energized equipment. It has also been a hot topic at industry conferences, which have highlighted strategies to reduce the risk of arc flash during testing of energized equipment.

Fixing the Problem, Not the Blame

Swensen said Facebook takes the issue of arc flash seriously, and wanted to share its findings with the industry.

“There are very important questions to ask,” he said. “If someone gets hurt because we didn’t take the right steps, what does that cost us as an organization?  It falls right in with Facebook’s concept with Open Compute – if we share this, it helps the industry.”

The 7×24 Exchange is known for lively question and answer sessions. At the Facebook/Schneider presentation, a questioner noted that challenges in busway assembly and maintenance are well understood, and wondered whether Facebook had considered alternate power designs.

“We have made very fundamental changes (in busway design),” since the incidents, Swensen said, but didn’t offer details.  Alternate methods include the use of cables, instead of busway, in power distribution.

Schneider Electric's Bill Westbrock speaks during the 7x24 Exchange Fall Conference in Phoenix. (Photo: Rich Miller)

Schneider Electric’s Bill Westbrock speaks during the 7×24 Exchange Fall Conference in Phoenix. (Photo: Rich Miller)

Schneider Electric and Facebook said the key to effective after-incident reviews is addressing the “blame game” up front.

“I would argue that all too often the filter your have in mind is ‘who’s to blame,’” said Swensen. “This is where, as end users, it’s important to set the right tone and culture.”

“A lot of people are protecting their interests,” said Westbrock. “We said ‘let’s not focus on assessing the blame, but understand where the breakdown occurred. James said ‘there will be no blame assessed and no financial compensation sought.’”

That won’t always be easy, especially in events involving outages or injuries. But Westbrock said the importance of the issue calls for nothing less.

“There are going to be problems as we build these projects,” said Westbrock. “We’re looking at critical problems. In this industry, we all need to own up to this to ensure that we do better.”

LinkedinTwitterFacebookSubscribe
Mail

Tagged With: 7x24 Exchange, Arc Flash, Facebook, Schneider Electric

Newsletters

Stay informed: Get our weekly updates!

Are you a new reader? Follow Data Center Frontier on Twitter or Facebook.

About Rich Miller

I write about the places where the Internet lives, telling the story of data centers and the people who build them. I founded Data Center Knowledge, the data center industry's leading news site. Now I'm exploring the future of cloud computing at Data Center Frontier.

Comments

  1. kwangmin.cho@navercorp.com'KwangMin Cho says

    October 19, 2017 at 9:08 pm

    Very nice article!! Thank you.

  2. rugis@rcn.com'Stan Rugis says

    October 23, 2017 at 7:37 am

    Not the first, nor the last sadly – electricians are sometimes ‘under the gun’ especially in the states. I always mention to the person doing the work, please take your time, then check – and if OT is needed to finish easier to pay $500 for the day the $500K later…

  3. pmuller@pmsolutions.org'Patrick Muller says

    October 24, 2017 at 7:53 am

    In my opinion, something simple was missed… thermal camera on all power connections, PDU, CDU, including busbar connexions , médium voltages and low voltage distributions during the commissioning (loaded! ) and annually as part of regular maintenance survey.
    Cost effective way to discover and plan preventive maintenances before accidents .

  4. dfaught@prismelectric.com'David Faught says

    December 31, 2017 at 10:39 pm

    Busway or cable both are subject to failure I have been in the business 30 years and have been involved with the repair of about one catastrophe failure every year. All Electrical and Mechincal systems require preventive maintenance and routine inspections. I have seen my electrical construction industry go from installations with no QAQC other than “good it turned on” to full time QAQC teams and commissioning of most projects by third parties. But none of these replace routine maintenance and inspections once the owner or property management takes over the property.

  • Facebook
  • Instagram
  • LinkedIn
  • Pinterest
  • Twitter

Voices of the Industry

Understanding the Differences Between 5 Common Types of Data Centers

Understanding the Differences Between 5 Common Types of Data Centers No two are data centers are alike when it comes to design or the applications and data they support with their networking, compute and storage infrastructure. Shad Secrist of Belden outlines the differences between 5 of the most common types of data centers including edge, colocation and hyperscale.

White Papers

data center provider

How to Choose a Data Center Provider

Asking the right questions is a big part of the equation. Here are eight considerations in a new white paper from Iron Mountain with targeted questions to ask that will help you choose the right colocation partner for your business. 

Get this PDF emailed to you.

We always respect your privacy and we never sell or rent our list to third parties. By downloading this White Paper you are agreeing to our terms of service. You can opt out at any time.

DCF Spotlight

Data center modules on display at the recent Edge Congress conference in Austin, Texas. (Photo: Rich Miller)

Edge Computing is Poised to Remake the Data Center Landscape

Data center leaders are investing in edge computing and edge solutions and actively looking at new ways to deploy edge capacity to support evolving business and user requirements.

An aerial view of major facilities in Data Center Alley in Ashburn, Virginia. (Image: Loudoun County)

Northern Virginia Data Center Market: The Focal Point for Cloud Growth

The Northern Virginia data center market is seeing a surge in supply and an even bigger surge in demand. Data Center Frontier explores trends, stats and future expectations for the No. 1 data center market in the country.

See More Spotlight Features

Newsletters

Get the Latest News from Data Center Frontier

Job Listings

RSS Job Openings | Pkaza Critical Facilities Recruiting

  • Critical Power Energy Manager - Data Center Development - Ashburn, VA
  • Site Development Manager - Data Center - Ashburn, VA
  • Data Center Facility Operations Director - Chicago, IL
  • Electrical Engineer - Senior - Dallas, TX
  • Mechanical Commissioning Engineer - Calgary, Alberta

See More Jobs

Data Center 101

Data Center 101: Mastering the Basics of the Data Center Industry

Data Center 101: Mastering the Basics of the Data Center Industry

Data Center Frontier, in partnership with Open Spectrum, brings our readers a series that provides an introductory guidebook to the ins and outs of the data center and colocation industry. Think power systems, cooling, solutions, data center contracts and more. The Data Center 101 Special Report series is directed to those new to the industry, or those of our readers who need to brush up on the basics.

  • Data Center Power
  • Data Center Cooling
  • Strategies for Data Center Location
  • Data Center Pricing Negotiating
  • Cloud Computing

See More Data center 101 Topics

About Us

Charting the future of data centers and cloud computing. We write about what’s next for the Internet, and the innovations that will take us there. We tell the story of the digital economy through the data center facilities that power cloud computing and the people who build them. Read more ...
  • Facebook
  • LinkedIn
  • Pinterest
  • Twitter

About Our Founder

Data Center Frontier is edited by Rich Miller, the data center industry’s most experienced journalist. For more than 20 years, Rich has profiled the key role played by data centers in the Internet revolution. Meet the DCF team.

TOPICS

  • 5G Wireless
  • Cloud
  • Colo
  • Connected Cars
  • Cooling
  • Cornerstone
  • Coronavirus
  • Design
  • Edge Computing
  • Energy
  • Executive Roundtable
  • Featured
  • Finance
  • Hyperscale
  • Interconnection
  • Internet of Things
  • Machine Learning
  • Network
  • Podcast
  • Servers
  • Site Selection
  • Social Business
  • Special Reports
  • Storage
  • Sustainability
  • Videos
  • Virtual Reality
  • Voices of the Industry
  • Webinar
  • White Paper

Copyright Data Center Frontier LLC © 2022