• About Us
  • Partnership Opportunities
  • Privacy Policy

Data Center Frontier

Charting the future of data centers and cloud computing.

  • Cloud
    • Hyperscale
  • Colo
    • Site Selection
    • Interconnection
  • Energy
    • Sustainability
  • Cooling
  • Technology
    • Internet of Things
    • AI & Machine Learning
    • Edge Computing
    • Virtual Reality
    • Autonomous Cars
    • 5G Wireless
    • Satellites
  • Design
    • Servers
    • Storage
    • Network
  • Voices
  • Podcast
  • White Papers
  • Resources
    • COVID-19
    • Events
    • Newsletter
    • Companies
    • Data Center 101
  • Jobs
You are here: Home / Cloud / Big Sur: A Closer Look at the Engine Powering Facebook’s AI

Big Sur: A Closer Look at the Engine Powering Facebook’s AI

By Rich Miller - July 13, 2016 Leave a Comment

Big Sur: A Closer Look at the Engine Powering Facebook’s AI

Racks packed with Big Sur servers inside the Facebook data center in Prineville, Ore. Each rack holds four of the GPU-accelerated chassis, which crunch data for artificial intelligence applications. (Photo: Facebook)

LinkedinTwitterFacebookSubscribe
Mail

PRINEVILLE, Ore. – In a row of racks deep inside a massive data center sits the engine of Facebook’s ambitions in artificial intelligence. Each server chassis is packed with computing horsepower, including graphics processing units (GPUs) that can crunch enormous volumes of data.

This is Big Sur, and it is the key to Facebook’s bid to  create a smarter newsfeed for its 1.6 billion users around the globe. Using Big Sur, Facebook can train its machine learning systems to recognize speech, understand the content of video and images, and translate content from one language to another.

Machine learning holds tremendous promise, but requires a lot of horsepower. The gains in performance and latency provided by Big Sur help Facebook crunch more data, dramatically shortening the time needed to train its neural networks.

Facebook has deployed thousands of Big Sur servers in its data centers, cireating a platform with 40 petaflops of computing power,  which would rank it among the world’s most powerful systems.

DIY Approach to Machine Learning

Each Big Sur unit consists of a 4U chassis, which houses eight NVIDIA M40 GPUs and two CPUs, along with SSD storage and hot-swappable fans at the rear of the server.

Neural networks mimic how the human brain works, allowing computers to adapt and “learn” tasks without being explicitly programmed for them. As leading tech companies push the boundaries of machine learning, they are often following a do-it-yourself approach.

Big Sur is a 4U chassis packed with eight NVIDIA M40 GPUs (Photo: Facebook)

Big Sur is a 4U chassis packed with eight NVIDIA M40 GPUs . The bright green ccover protects the motherboard and CPUs, but also assists in cooling. (Photo: Facebook)

“There wasn’t any good hardware on the market at the time, so we developed our own,” said Kevin Lee, a Technical Program Manager for Facebook. They reached out to NVIDIA, whose graphcs processing units were initially focused on accelerating video games, but now have seen broad adoption in high-performance computing.

“We work very closely with Facebook Research,” said Ian Buck, Vice President of Accelerated Computing at NVIDIA.

Getting Parallel, Getting Results

Using CPUs, it took three months to train a machine learning neural network. Adding GPUs reduced that processing time to a month, Buck said.

Facebook and NVIDIA began working together to optimize a design using the NVIDIA Tesla M40 GPU accelerator, the company’s flagship product for deep learning.

Free Resource from Data Center Frontier White Paper Library

cloud service providers
Pro Tips and Best Practices: Physical Layer Strategies for Cloud/Managed Service Providers
Successful Cloud Service Providers and Managed Service Providers need to be out in front of everything in their managed data center spaces – ensuring uptime, bandwidth, and operational/cost efficiency today, with the flexibility and scalability to adapt and expand on the fly. Physical layer and  infrastructure is the foundation on which those services are built. Get the new data center ebook from Siemon that explores pro tips and best practices for physical layer strategies for cloud and managed service providers, from zone cabling in the colocation data center to high speed interconnects in the data center.
We always respect your privacy and we never sell or rent our list to third parties. By downloading this White Paper you are agreeing to our terms of service. You can opt out at any time.

Get this PDF emailed to you.

Each M40 GPU houses 3,072 NVIDA CUDA computing cores, small processors that work together on computing tasks. By working in parallel with many processors at once, GPUs can accelerate many types or workloads.

“With Big Sur, you can now train a neural network in less than a day,” said Buck.

Facebook has optimized these new servers for thermal and power efficiency, allowing them to operate alongside standard CPU-powered servers in Facebook’s data centers. In Prineville, the Big Sur cluster shares space with Open Compute servers inside PRN2, a 350,000 square foot server farm that is longer than an aircraft carrier.

A view of Big Sur with the cover removed, exposing the motherboard, CPUs and DIMM memory. (Photo: RIch Miller)

A view of Big Sur with the cover removed, exposing the motherboard, CPUs and DIMM memory. (Photo: RIch Miller)

Each Tesla GPU can use up to 300 watts of electricity, bringing the power footprint for each Big Sur chassis to about 2.5 kilowatts. Facebook likes to keep power densities below 12 kW a rack, so it houses four Big Sur units in each rack, resulting in a power footprint of 10kW per rack.

Like most Open Compute designs, Big Sur’s components are arranged to optimze airflow through the chassis. Each Big Sur unit features a bright green plastic over over the motherboard and processors. This protects the components, but is also designed to enhance the airflow, narrowing near the back to create a “Venturi effect” that accelerates the air movement across the GPUs.

Smarter Services Through AI

Facebook isn’t alone in prioritizing artificial intelligence and machine learning. Google, Apple and Amazon have also created research labs to pursue faster and better AI capabilities. They have used different approaches to hardware, with Google opting for custom ASICs (application specific integrated circuits) for its machine learning operations.

Whatever the hardware approach, the goal is the same: take elements of digital assistants like Apple’s Siri and the Amazon Echo and build them into almost every element of their technology and services.

Facebook’s Lee said the company is already using Big Sur to train neural networks to “read” to the blind, recognizing images and sharing a spoke description of their contents. Buck and Lee demonstrated how Facebook;s AI can review thousands of paintings, recognize desired visual elements in the painting, and incorporate them into a new work of art

In the same way, Facebook’s servers and algorithms can learn from the content shared by its users, and tailor the updates it selects for each person’s newsfeed. Artifical intelligence aligns with Facebook’s culture of rapid iteration, constantly testing new approaches to improving its products.

“The whole point is to go fast,” said Ken Patchett, Facebook’s Director of Data Center Operations at Prineville. “We have to get better quickly.”

LinkedinTwitterFacebookSubscribe
Mail

Tagged With: Artificial Intelligence, Facebook, NVIDIA

Newsletters

Stay informed: Get our weekly updates!

Are you a new reader? Follow Data Center Frontier on Twitter or Facebook.

About Rich Miller

I write about the places where the Internet lives, telling the story of data centers and the people who build them. I founded Data Center Knowledge, the data center industry's leading news site. Now I'm exploring the future of cloud computing at Data Center Frontier.

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

  • Facebook
  • Instagram
  • LinkedIn
  • Pinterest
  • Twitter

Voices of the Industry

Fit for the Edge: Modular Data Centers

Fit for the Edge: Modular Data Centers Jackie Pasierbowicz, Director of Sales and Marketing at TAS explores the dramatic rise of multiple markets for edge computing and the benefits of a modular data center strategy.

DCF Spotlight

The COVID-19 Crisis and the Data Center Industry

The COVID-19 pandemic presents strategic challenges for the data center and cloud computing sectors. Data Center Frontier provides a one-stop resource for the latest news and analysis for decision-makers navigating this complex new landscape.

An aerial view of major facilities in Data Center Alley in Ashburn, Virginia. (Image: Loudoun County)

Northern Virginia Data Center Market: The Focal Point for Cloud Growth

The Northern Virginia data center market is seeing a surge in supply and an even bigger surge in demand. Data Center Frontier explores trends, stats and future expectations for the No. 1 data center market in the country.

See More Spotlight Features

White Papers

COVID-19 IT Response

COVID-19 IT Response & Recovery Report

Thirty-seven percent of IT professionals surveyed said that their spending was cut in the short-term during the early months of the pandemic. But the good news is that 2021 spending outlooks shows signs of rebounding. Learn how IT professionals nationwide responded, repositioned and are rebuilding in light of COVID-19, via a new report from Service Express.

Get this PDF emailed to you.

We always respect your privacy and we never sell or rent our list to third parties. By downloading this White Paper you are agreeing to our terms of service. You can opt out at any time.

Newsletters

Get the Latest News from Data Center Frontier

Job Listings

RSS Job Openings | Peter Kazella and Associates, Inc

  • Navy Electrician / Navy Mechanic - Redmond, WA
  • Electrical Commissioning Engineer - Ashburn, VA
  • MEP Superintendent - Data Center - Dallas, TX
  • Construction Project Manager - Data Center - Dallas, TX
  • Data Center QA / QC Manager - Huntsville, AL

See More Jobs

Data Center 101

Data Center 101: Mastering the Basics of the Data Center Industry

Data Center 101: Mastering the Basics of the Data Center Industry

Data Center Frontier, in partnership with Open Spectrum, brings our readers a series that provides an introductory guidebook to the ins and outs of the data center and colocation industry. Think power systems, cooling, solutions, data center contracts and more. The Data Center 101 Special Report series is directed to those new to the industry, or those of our readers who need to brush up on the basics.

  • Data Center Power
  • Data Center Cooling
  • Strategies for Data Center Location
  • Data Center Pricing Negotiating
  • Cloud Computing

See More Data center 101 Topics

About Us

Charting the future of data centers and cloud computing. We write about what’s next for the Internet, and the innovations that will take us there. We tell the story of the digital economy through the data center facilities that power cloud computing and the people who build them. Read more ...
  • Facebook
  • LinkedIn
  • Pinterest
  • Twitter

About Our Founder

Data Center Frontier is edited by Rich Miller, the data center industry’s most experienced journalist. For more than 20 years, Rich has profiled the key role played by data centers in the Internet revolution. Meet the DCF team.

TOPICS

  • 5G Wireless
  • Cloud
  • Colo
  • Connected Cars
  • Cooling
  • Cornerstone
  • Coronavirus
  • Design
  • Edge Computing
  • Energy
  • Executive Roundtable
  • Featured
  • Finance
  • Hyperscale
  • Interconnection
  • Internet of Things
  • Machine Learning
  • Network
  • Podcast
  • Servers
  • Site Selection
  • Social Business
  • Special Reports
  • Storage
  • Sustainability
  • Videos
  • Virtual Reality
  • Voices of the Industry
  • White Paper

Copyright Data Center Frontier LLC © 2021